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Preface 


Evaluation  is  an  integral  part  of  the  management  of  the  Nutrition  Education  and  Training 
Program  (NET).  The  purpose  of  this  Guide  is  to  assist  the  State  NET  Coordinator  in  planning 
and  managing  the  evaluation  component  of  the  State  Plan  as  required  in  the  NET  regulations 
(7  CFR  227.37(b)(l4)). 

Evaluation  is  a  creative  process  and  there  are  many  different  approaches  to  evaluation  design  and 
methodology.  The  authors  and  reviewers  of  this  Guide  had  to  make  decisions  about  what  to 
include  in  the  Guide  and  how  to  describe  an  approach  to  evaluation  for  use  by  State  NET 
Coordinators.  The  approach  to  evaluation  described  here  is  used  by  some  experts  in  the  field, 
however,  there  are  other  valid  approaches  to  evaluation  and  other  useful  methods  and  techniques. 
The  NET  Coordinators  and  other  readers  are  encouraged  to  design  evaluation  using  this  Guide  as 
well  as  other  references  and  experts  who  are  available  for  consultation  on  specific  aspects  of  evalu- 
ation. To  design  effective  and  efficient  evaluation,  the  NET  Coordinator  must  be  sensitive  to  the 
information  needs  and  resource  constraints  of  the  NET  Program  and  the  individual  projects  for 
which  evaluation  is  designed. 

This  Guide  is  written  as  a  companion  document  to  the  NeeiJs  Assessment  Guide  For  The  Nutrition 
Education  And  Training  Program  (1994).  Because  needs  assessment  and  evaluation  are  so  closely 
related  in  program  design  and  management,  there  are  necessarily  many  areas  of  common  content 
in  the  two  Guides.  To  the  extent  possible,  repetition  of  content  in  the  NET  Evaluation  Guide  has 
been  eliminated  by  referencing  appropriate  parts  of  the  NET  Needs  Assessment  Guide.  Therefore,  it 
is  recommended  that  the  reader  have  a  copy  of  the  NET  Needs  Assessment  Guide  easily  available  for 
reference. 
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Chapter  I 
Introduction  to  Evaluation 

Purpose 

The  purpose  of  this  Evaluation  Guide  is  to  provide  guidance  to  State  agencies  for  designing  the 
evaluation  component  of  the  Nutrition  Education  and  Training  (NET)  State  Plan  required  by 
Food  and  Consumer  Service  (ECS)  regulations  and  defined  by  the  Strategic  Plan  for  Nutrition 
Education  (1993).  The  use  of  a  comprehensive  evaluation  model  as  described  in  this  Guide  can 
enhance  the  effectiveness  of  a  State's  Child  Nutrition  Program. 

Overview  of  the  Guide 

This  Guide  has  been  developed  to  assist  the  NET  Coordinator  in  planning  and  managing  the 
evaluation  of  the  NET  State  Plan  which  details  the  individual  projects  and  activities  funded 
through  NET.  It  is  recommended  that  the  Guide  be  used  as  a  reference  along  with  the  NET  Needs 
Assessment  Guide  (1994).  This  Guide  begins  with  an  explanation  of  evaluation  and  reasons  for  pro- 
gram evaluation  in  Chapter  I,  Introduction  to  Evaluation.  Chapter  II,  Developing  and  Managing 
Evaluation,  describes  how  to  develop  an  evaluation  plan  for  the  NET  State  Plan.  Chapter  III,  The 
CIPPI  Model  for  Evaluation,  presents  one  of  the  most  commonly  used  frameworks  for  the  design  of 
a  comprehensive  evaluation  plan.  Subsequent  chapters  provide  detailed  information  on  how  to  use 
the  Model  for  NET  evaluation.  Chapter  IV,  Application  of  the  CIPPI  Model  -  NET  Evaluation  Plan, 
provides  an  example  of  the  evaluation  component  of  some  State  Plan  objectives.  Chapter  V,  pro- 
vides an  Explanation  of  Context  Evaluation.  An  Explanation  of  Input  Evaluation  is  provided  in 
Chapter  VI.  Chapter  VII  is  an  Explanation  of  Process  Evaluation.  An  Explanation  of  Product 
Evaluation  is  provided  in  Chapter  VIII  with  an  Explanation  of  Impact  Evaluation  in  Chapter  IX. 

It  is  recommended  that  the  reader  review  Chapter  IV,  Application  of  the  CIPPI  Model  -  NET 
Evaluation  Plan  before  going  further  in  this  Guide.  This  example  of  an  evaluation  plan  for  some 
NET  State  Plan  objectives  illustrates  the  evaluation  model  described  in  this  Guide  and  will  make 
the  content  more  meaningful  to  the  reader. 

What  is  Evaluation? 

Evaluation  of  the  NET  Program  involves  determining  the  worth  or  value  of  specific  resources  and 
activities  in  meeting  the  needs  for  which  the  Program  is  intended.  Evaluation  could  be  designed 
to  assess: 

S  the  State  NET  Program  as  a  whole, 

S  a  food  and  nutrition  training  program  component  of  the  State  Plan,  or 
®  a  nutrition  education  project  component  of  the  State  Plan. 

The  worth  or  value  of  the  Program  is  determined  through  quantitative  and  qualitative  measure- 
ments of  program  services  including  the  quality  of  the  services  provided  and  the  success  of  the 
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services  in  achieving  the  desired  outcomes.  The  results  of  evaluation  are  used  for  policy  and  man- 
agement decision  making  and  for  program  accountability  (Stufflebeam,  1971;  Wholey,  1991). 

Program  evaluation  mav  be  viewed  as  a  prospective  or  a  retrospective  approach.  A  NET  evaluation  for 
decision  making  could  be  considered  a  prospective  approach,  in  this  case  it  is  essentially  a  needs 
assessment  process.  The  information  gathered  will  relate  to  the  future  and  be  used  to  answer 
questions  about  what  the  NET  Program  should  be  doing  and  how  the  services  should  be  deliv- 
ered. For  example,  the  evaluation  of  a  statewide,  elementan,-  nutrition  education  program  will 
provide  valuable  information  about  the  design  of  future  elementary  nutrition  education  pro- 
grams. 

A  NET  evaluation  designed  for  accountability  could  be  considered  a  retrospective  approach,  or  a 
look  back  at  things  past.  Here,  the  evaluator  is  looking  back  on  decisions  that  were  made  and  the 
resources  and  activities  offered  to  determine  if  the  defined  needs  were  met.  An  evaluation  of  a 
NET  Program  that  addresses  whether  the  State  agency  is  making  a  difference  over  the  long  term, 
provides  data  for  accountability. 

Role  of  the  NET  Coordinator 

As  the  manager  of  the  State  NET  Program,  the  NET  Coordinator  is  involved  in  ever\^  aspect  of 
the  Program  from  needs  assessment  to  project  management  to  program  evaluation.  Evaluating 
the  program  goals  and  the  specific  projects  funded  by  the  NET  Program  is  one  of  the  most  chal- 
lenging aspects  of  the  NET  Coordinator's  role. 

"Why  Evaluate? 

To  effectively  manage  a  State  NET  Program,  consideration  should  be  given  to  the  following  rea- 
sons for  program  evaluation. 

1.  NET  regulations  require  evaluation  to  he  addressed  in  the  State  Plan. 
FNS  regulations  7  CFR  Part  227  describe  the  implementation  and  operation  requirements  of  the 
NET  Program.  The  regulations  (Section  227.37(b)  (14))  specify  that  the  NET  State  Plan  shall 
include,  "...plans  to  evaluate  program  activities  including  an  evaluation  component  for  each 
objective  of  the  State  Plan."  Program  evaluation  must  be  addressed  in  order  to  effectively  manage 
the  Program  and  to  use  the  available  funds  for  the  maximum  impact  in  the  State.  The  NET  regu- 
lations describe  the  required  evaluation: 

227.31(c)  'Evaluations.  The  State  agency  shall  conduct  formal  evaluations  of  program  activities 
at  least  annually.  These  evaluations  shall  be  aimed  at  assessing  the  effectiveness  of  the  various 
activities  undertaken  by  the  State  and  local  agencies.  State  officials  shall  analyze  why  some 
activities  have  proved  effective  while  others  have  not  and  shall  initiate  appropriate  improve- 
ments. Representatives  of  State  Advisor^'  Councils  and  agencies  that  participate  in  the  State 
agencies'  programs  as  well  as  representatives  of  other  groups  shall  be  invited  to  participate  in 
the  evaluations.  The  results  of  the  evaluations  shall  be  used  to  make  adjustments  in  ongoing 
activities  and  to  plan  activities  and  programs  for  the  next  year's  State  plan.  The  State  agency 
shall  submit  a  plan  for  evaluation  of  Program  activities  as  part  of  the  State  plan  in  accordance 
with  227.37(b)(l4). 
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2.  The  Strategic  Plan  For  Sutrition  Education  (1993)  defines  evaluation  as  a  basis  for  pro- 
gram development,  improvement,  and  accountability.  The  Nutrition  and  Technical  Services, 
Food  and  Nutrition  Sen,-ice  (now  Food  and  Consumer  Service),  USDA,  initiated  a  strategic  plan- 
ning process  for  the  Nutrition  Education  and  Training  Program  in  March  1992.  The  participants 
in  the  strategic  planning  process  defined  a  vision  and  strategic  direction  for  the  NET  Program 
that  provides  a  clear  and  concise  conception  of  the  program's  future.  The  strategic  direction  helps 
guide  program  planners  as  they  make  decisions,  "...regarding  organizational  policy,  strategy,  pro- 
grams, and  resource  allocations."  (Mandell,  1993).  The  Strategic  Direction  addresses  a  State's 
responsibilit}'  for  conducting  "...formal  needs  assessments  and  evaluations  as  the  basis  for  pro- 
gram development,  improvement,  and  accountabilit}-."  The  ten  goals  included  in  the  Strategic 
Plan  are  national  planning  goals  that  should  be  used  by  States  to  develop  their  own  goals  as 
defined  by  the  identified  needs  of  the  targeted  populations  (See  Appendbc  F). 

3.  Evaluation  data  should  be  used  to  make  decisions  for  program  improvement.  Assessment 
data  provides  the  NET  Coordinator  with  information  on  which  to  base  program  planning  and 
design  decisions,  while  evaluation  data  provides  information  for  program  improvement  and 
accountability^  Program  goals  and  objectives  provide  the  basis  on  which  to  measure  program 
improvement. 

4.  Evaluation  can  help  to  show  cost-effectiveness.  Evaluation  data  can  be  used  to  help  justify 
funding  for  the  NET  Program  and  other  Child  Nutrition  Programs. 

3.  Evaluation  enhances  program  credibility.  During  a  time  of  increasing  need  and  decreasing 
resources,  professionals  and  the  general  public  question  the  significance  of  service  programs. 
Reports  of  NET  Program  performance  can  enhance  the  credibility  of  the  program  within  the 
State  agenq,'  and  the  department,  to  polic}-  makers  as  well  as  to  program  stakeholders. 

6.  Evaluation  supports  the  NET  Coordinator's  professionalism.  Seeing  the  results  of  one's 
efforts  promotes  a  sense  of  self-satisfaction.  Knowing  that  one's  management  skills  and  profes- 
sionalism contributed  to  the  success  of  a  program  increases  job  satisfaction.  Professional  managers 
can  justif}"  their  position,  show  the  value  of  their  job  function,  and  demonstrate  their  contribu- 
tion to  the  organization  through  a  well-designed  evaluation  of  program  objectives.  Coordinators 
can  demonstrate  the  contributions  made  by  NET  to  the  overall  Child  Nutrition  Programs  and  to 
the  education  programs  in  their  State. 

The  Evaluation  Challenge 

Evaluation  is  one  of  the  most  challenging  aspects  of  program  management.  Too  often,  program 
managers  avoid  this  crucial  aspect  of  program  management  because  of  concerns  stemming  from  a 
lack  of  understanding  about  the  evaluation  process  itself.  To  learn  how  to  plan  and  manage  the 
evaluation  of  the  NET  State  Plan  objectives,  universal  concerns  need  to  be  acknowledged. 
Following  is  a  discussion  of  some  of  the  concerns  which  NET  Coordinators  may  have. 
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Concern  #1."  "'^th  limited  funding,  there  is  not  enough  money  to  evaluate  the  program." 

This  concern  is  a  common  one  in  most  sen-ice  programs.  A  well-designed  evaluation  does  not 
have  to  be  an  expensive  undertaking.  The  issue  of  limited  resources  for  evaluation  can  be  resolved 
by  including  evaluation  as  an  integral  part  of  the  design  of  the  entire  program,  not  isolating  eval- 
uation as  the  expendable  component  of  the  program  when  cuts  have  to  be  made.  Since  evaluation 
provides  data  for  both  decision  making  and  program  accountability',  without  evaluation  the  pro- 
gram cannot  be  expected  to  meet  the  needs  of  the  targeted  groups  nor  conform  to  program 
requirements  over  the  long  term.  The  question  to  be  answered  should  be,  "How  should  the  eval- 
uation be  designed?'"  rather  than.  "Should  evaluation  be  undertaken?" 

Concern  #2;  "As  a  program  planner,  I  don't  know  what  information  to  collect."  The  pur- 
pose of  evaluation  is  to  determine  to  what  extent  goals  and  objectives  have  been  accomplished. 
Thus,  clearly  defined  objectives  for  the  program  provide  the  basis  for  the  evaluation.  When 
objectives  are  not  clear,  the  program  planner  does  nor  know  what  information  to  collect. 

Concern  #5/  "There  are  too  many  variables  affecting  the  outcomes  for  me  to  isolate  the 
effects  of  the  program."  "With  any  aspect  of  a  food  service  training  program  or  a  nutrition  edu- 
cation program,  there  are  certainly  many  variables  which  affect  the  outcomes.  For  example,  when 
training  school  food  service  managers  to  implement  the  Dietary  Guidelines  in  their  schools,  a  few 
of  the  variables  which  impact  their  ability  to  do  so  include:  the  on-going  support  and  knowledge 
of  the  district  school  food  service  direaors;  the  support  of  the  school  administration  and  faculty; 
the  support  and  interest  of  the  parents  and  community;  the  management  skills  of  the  manager; 
the  availability  of  food  items  which  are  consistent  with  the  Dietary  GuideHnes;  and  the  knowl- 
edge and  skills  of  the  food  service  assistants.  "Waiting  for  program  operations  to  become  simpler 
is  not  reasonable.  If  anything,  most  of  the  work  involved  in  the  operation  of  Child  Nutrition 
Programs  becomes  more  complicated  each  year.  Evaluation  will  not  be  as  precise  as  that  used  for 
a  controlled  experimental  project,  but  an  organized  approach  to  evaluation  that  acknowledges 
lack  of  control  of  some  variables  is  preferable  to  no  evaluation  at  all. 

Concern  #4/  "Program  evaluation  may  lead  to  information  we  would  rather  not  have." 
Good  evaluation  design  provides  an  opportunity'  to  obtain  varying  perspectives  of  program 
implementation.  It  is  certainly  possible  that  the  evaluation  data  may  reveal  difficulty  in  accom- 
plishing objectives,  just  as  it  reveals  success.  Impact  evaluation  of  a  long-term  training  program 
may  tell  the  evaluator  that  participants  did  not  actually  change  their  behavior  on  the  job.  Input 
evaluation  of  a  nutrition  education  program  may  reveal  that  lack  of  teacher  training  severely  lim- 
ited the  use  of  a  State  curriculum  guide  in  elementary  classrooms.  Knowing  more  about  the 
implementation  and  results  of  a  program  provides  critical  information  for  long-term  program 
improvement  and  for  maximum  use  of  available  resources. 

Summary 

Program  evaluation  is  basic  to  the  effective  management  of  a  State  NET  Program.  Evaluation  is 
required  in  the  NET  regulations  as  a  measure  of  program  accountability  and  is  necessary  for  man- 
agement decision  making.  Planning  and  implementing  evaluation  is  an  integral  part  of  NET 
Program  management. 
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Chapter  II 
Developing  and 
Managing  Evaluation 


Evaluation  as  Part  of  the  A-P-B-I-E  Process 

The  NET  Needs  Assessment  Guide  (1994)  described  NET  program  management  as  the  Closed  Loop 
Assessment-PIanning-Budgering-Implementation-Evaluation  Process  (A-P-B-l-E  Process).  This 
process  illustrates  how  evaluation  is  an  integral  part  of  program  management.  Although  the 
results  of  evaluation  closes  the  loop  in  the  management  process,  planning  for  evaluation  begins 
when  the  State  Plan  is  developed.  A  systematic  approach  to  planning  and  implementing  the  eval- 
uation aspect  of  the  NET  Program  is  essential  to  obtaining  and  using  evaluation  data  to  improve 
the  Program. 


The  needs  assessment  provides  the  futuristic  perspective  for  the  process.  As  the  initiating  com- 
ponent, the  primary'  focus  of  needs  assessment  is  the  determination  of  "where  are  we  now  in 
relation  to  where  we  want  to  go?"  "What  are  our  greatest  challenges  today  and  what  chal- 
lenges do  we  project  for  tomorrow?"  "What  needs  must  be  addressed  if  we  are  to  achieve  our 
goals?"  "What  obstacles  must  be  overcome?"  "How  will  that  impact  be  felt?"  "What  should 
be  our  priorities?"  The  answers  to  those  questions  provide  the  basis  for  planning,  including 
the  identification  of  specific  objectives  to  be  accomplished  for  a  given  period  of  time  and  the 
outline  of  the  action  plan  which  will  contribute  to  the  achievement  of  the  objectives. 
Answers  also  provide  the  basis  for  all  subsequent  resource  allocations  ( budgeting)  to  be  made 
during  that  time-frame. 

Following  implementation  of  the  State  Plan,  the  evaluation  assumes  a  historical  focus  by  return- 
ing to  the  same  questions  asked  in  the  needs  assessment,  "After  all  our  effort,  where  are  we 
NOW  in  relation  to  where  we  want  to  go?"  "What  didn't  work  and  why?"  "What  impact 
did  we  have  in  addressing  our  defined  needs?"  "Were  our  priorities  appropriate  relative  to 
our  current  conditions  and  to  the  changes  taking  place  in  our  State?"  The  answers  to  those 
questions,  in  turn,  provide  the  basis  for  a  re-examination  or  re-assessment  of  projected  needs. 
And,  thus,  the  cycle  begins  anew. 

The  assessment  and  evaluation  components  represent  both  the  initial  and  final  phase  of  a 
closed  loop  assessment -planning-budgeting-implementation-evaluation  (A-P-B-I-E)  process 
or  cycle. 

{NET  Needs  Assessment  Guide,  1994,  pp.  10-11) 
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Illustration  1.  Closed  Loop  Assessment-Planning -Budgeting-Implementation-Evaluation  (A-P-B-I-E)  Process. 
Source:  NET  Needs  Assessment  Guide  ( 1994),  p.lO 

Since  evaluation  is  the  final  step  in  the  Closed  Loop  A-P-B-I-E  Process,  it  brings  the  program 
manager  full  circle,  back  to  the  questions  raised  in  the  needs  assessment  process  when  obstacles 
that  must  be  overcome  are  identified,  and  needs  that  must  be  addressed  to  meet  the  goals  and 
objectives  of  the  NET  Program  are  defined.  Using  the  results  from  evaluation,  the  Coordinator 
can  make  the  decision  to  repeat  a  program  initiative  that  was  successful,  to  develop  program 
improvements  for  an  initiative,  or  to  eliminate  a  program  initiative  in  favor  of  a  different 
approach  to  meeting  identified  needs.  Developing  an  evaluation  plan  is  essential  to  obtaining 
data  necessary  to  make  program  management  decisions. 
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Planning  for  Evaluation 


The  plan  for  NET  evaluation  must: 

•  Be  designed  as  part  of  the  State 
Plan  development  process  to 
coordinate  with  the  objectives 
defined  in  the  State  Plan: 

•  Be  easily  understood  and  used; 

•  Provide  for  a  constantly  evolving, 
ever-changing,  continuous  process 
of  evaluation; 

•  Be  perceived  as  purposeful; 

•  Include  a  varien.'  of  types  of 
evaluation  that  address  various 
aspects  of  planning  and 
implementation; 

•  Provide  a  basis  for  improved  prac- 
tices and  program  management;  and 

•  Promote  objectivity  in  decision 
making. 


De\'eloping  a  plan  for  NET  evaluation  involves  sev- 
eral important  decisions.  Since  evaluation  is  a  con- 
tmuous  process,  the  decisions  should  be  reconsid- 
ered periodically.  Decisions  which  influence  the 
overall  evaluation  plan  involve  determining: 
A  balance  between  the  use  of  formative  and 
summative  evaluation, 
^  A  balance  between  formal  and  informal  evalua- 
tion design, 

^  A  balance  between  the  use  of  quantitative  and 
qualitative  methods,  and 
Fair  evaluation  in  relation  to  proximity  of  out- 
comes. 


^   A  balance  betueen  the 
use  of  formative  and 
summative  evaluation 


When  planning  the  evaluation  phase  of  the  A-P-B-I-E  cycle,  two 
questions  shoxild  be  addressed: 

''"When  should  the  design  of  the  evaluation  process  begin?"  and 

'"'When  should  the  data  collection  aspect  of  the  evaluation  process  begin?" 


There  is  no  evidence  of  debate  among  evaluation  experts  as  to  the 
answer  to  the  first  question.  The  plan  for  evaluation  must  be 
developed  during  the  overall  program  planning  process,  since  eval- 
uation design  is  an  integral  part  of  planning.  Evaluation  questions 
(or  evaluation  objectives)  must  parallel  program  objectives  if  eval- 
uation is  to  determine  the  progress  made  by  the  program  or  pro- 
ject, and  guide  future  planning  decisions.  Specific  evaluation  ques- 
tions should  be  defined  for  each  objective  in  the  State  Plan.  For 
example,  if  the  State  Plan  includes  an  objective  to  develop  a  pro- 
gram to  positively  influence  the  food  choices  made  at  school  by 
middle  school  students,  the  evaluation  plan  must  answer  the  ques- 
tion, "To  what  extent  did  the  program  make  a  positive  differ- 
ence?" If  the  State  Plan  addresses  the  preparation  of  K-3  teachers 
to  select  appropriate  nutrition  education  materials  and  use  them 
eflFectively  in  their  instruction,  the  evaluation  question  to  be 
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^    A  balance  between  the  answered  is,  "How  appropriately  can  teachers  make  material  selec- 

use  of  formative  and  tions  and  use  them  eftectively  in  their  nutrition  education  instruc- 

summative  evaluation  tion?'  How  data  will  be  collected  to  answer  these  questions  must 

{Continued}  be  determined  as  the  plan  is  developed. 

The  answer  to  the  second  question,  "  When  should  the  data  collec- 
tion aspect  of  the  evaluation  process  begin?"  depends  on  how  the 
evaluation  data  will  be  used.  The  data  collection  should  begin 
with  the  initiation  of  the  overall  program  planning  process  when 
information  is  needed  for  monitoring  the  progress  of  the  program 
and  making  adjustments  to  promote  program  success  (formative 
evaluation).  When  evaluation  information  will  be  used  for  pro- 
gram accountability  and  for  making  planning  decisions  that  affect 
the  next  cycle  of  the  program,  the  data  should  be  collected  as  the 
projects  and  activities  are  completed  (summative  evaluation). 

Consider  the  example  of  a  six-week  staff  development  program  for 
health  educators  for  which  a  new  training  workbook  has  been 
developed.  Formative  evaluation  needs  to  be  included  within  the 
overall  evaluation  design  of  the  staff  development  program  and 
data  should  be  collected  during  the  implementation  phase  of  the 
program.  With  this  approach,  the  NET  Coordinator  might  find 
that  the  post-tests  given  at  the  end  of  sessions  two  and  three  indi- 
cate low  scores  among  the  participants.  Obviously,  mid-program 
corrections  need  to  be  made,  but  without  the  formative  evaluation 
data  provided  by  the  post-tests,  the  Coordinator  would  not  know 
that  program  adjustments  were  needed.  With  these  data,  the 
Coordinator  can  make  decisions  about  what  changes  should  be 
made. 

Summative  evaluation  should  also  be  included  as  part  of  the  evalu- 
ation design.  This  will  involve  collecting  data  after  the  health 
educators'  staff  development  program  has  been  completed.  These 
data  should  be  used  to  make  decisions  about  whether  the  staff 
development  program  for  health  educators  should  be  continued  as 
is,  terminated,  offered  to  more  teachers  or  fewer  teachers,  or 
offered  to  different  categories  of  teachers.  Additionally,  the  sum- 
mative evaluation  data  are  needed  for  program  accountability  and 
for  justification  of  the  expendinare  of  NET  funds. 
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.4  balance  between 
formal  and  informal 
evaluation  design 


^    A  balance  betueen 
quantitative  and 
qualitative  evaluation 
methods 


The  formality  of  evaluation  design  can  be  placed  on  a  continuum. 
Based  on  available  time,  budget,  and  expenise,  an  e\'aluation  plan 
may  be  quite  formal  and  highly  structured.  When  taken  to  an 
extreme  degree  of  sophistication,  an  evaluation  can  become  so 
cumbersome,  time-consuming,  and  expensive  that  the  cost  of  pro- 
ducing the  findings  cannot  be  offset  by  subsequent  program 
improvements. 

At  the  other  extreme  of  the  formality'  continuum  is  the  collection 
of  informal  information  based  on  a  "how  do  you  feel  and  what  do 
you  think"  five-question  form  given  at  the  conclusion  of  a  three- 
hour  seminar.  Though  this  information  has  its  purpose,  too  often 
this  is  the  sum  and  substance  of  an  evaluation  effort.  Because  the 
purpose  of  evaluation  is  to  determine  the  contribution  of  programs 
and  activities  to  the  overall  success  of  the  NET  Program,  substan- 
tive data  must  be  collected.  If  the  evaluation  is  so  "slap-stick"  and 
informal  that  the  results  are  meaningless,  there  will  be  no  data 
base  for  determining  program  accountability-  nor  for  making  pro- 
gram improvement  decisions. 

Another  continuum  relating  to  evaluation  design  addresses  the 
issue  of  quantitative  or  empirical  research  at  one  end,  and  qualita- 
tive research  on  the  other.  Achterberg  (1988)  finds  that,  tradition- 
ally, nutrition  education  has  focused  primarily  on  quantitative 
evaluation.  Quantitative  evaluation  seeks  to  determine  effective- 
ness by  use  of  pre-  and  post-assessment  measures.  It  typically 
includes  evaluation  designs  and  techniques  that  produce  discrete, 
quantifiable  data  which  can  then  be  sxibjected  to  rigorous  statisti- 
cal treatment.  Achterberg  argues  that  exclusive  use  of  this 
methodology  fails  to  provide  the  decision-maker  with  any  data  on 
how  or  why  change  occurred  or  failed  to  occur. 

Qualitative  evaluation,  in  contrast,  fociises  on  an  explanation  of 
"why"  and  "the  meaning  behind"  an  occurrence  or  outcome  rather 
than  differences  of  statistical  significance.  It  provides  more  com- 
plete information  on  the  subtleties  that  contribute  to  differences 
found  when  comparing  pre-and  post-measurements.  It  can  also 
provide  valuable  information  on  effects,  happenings,  and  changes 
which  were  not  anticipated  in  the  original  evaluation  design,  as 
well  as  information  on  subjective  phenomena  which  cannot  be 
objectively  observed  or  measured.  Qualitative  evaluation  uses 
methods  such  as  open-ended  inter\'iews,  obser\"ations.  extracts 
from  documents,  and  expert  judgments  to  assess  "hard  to  mea- 
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A  balance  hetueen         sure "  phenomena  (Wholey,  1991).  One  should  be  aware  chat 
quantitative  and  although  qualitative  evaluation  provides  a  valuable  perspective  of 

qualitative  evaluation    the  total  picture  of  a  program  or  project,  the  analysis  of  qualitative 


methods 
{Continued ) 


data  can  often  be  labor  intensive. 

When  considering  whether  to  use  qualitative  evaluation  designs  in 
lieu  of  quantitative  designs.  Tukey  states,  "it  is  better  to  have  an 
approximate  answer  to  the  right  question,  which  is  often  vague, 
than  an  exact  answer  to  the  urmg  question,  which  can  always  be 
made  more  precise."  (Lincoln  and  Guba,  1985).  Filstead  calls  for 
the  use  of  both  designs,  suggesting  that  the  qualitative  methods 
"provide  the  context  of  meanings  in  which  the  quantitative  meth- 
ods can  be  understood."  (19~9). 

It  is  doubtfiil  that  the  excessive  use  of  either  type  of  evaluation 
can  provide  the  array  of  information  needed  to  make  wise  deci- 
sions. Rather,  Achterberg  and  others  encourage  the  use  of  a  com- 
bination of  qualitative  and  quantitative  methods  '"so  that  the  flaws 
of  one  method  can  be  compensated  for  by  the  strengths  of  another 
method."  (1988). 


^   Fair  evaluation  in 

relation  to  proximity 
of  outcomes 


An  issue  in  designing  evaltiation  for  nutrition  education  programs 
involves  determining  fair  evaluation  in  relation  to  the  proximity 
of  the  outcomes  of  the  program  or  project  to  the  actual  implemen- 
tation (St. Pierre,  1982).  For  example,  when  a  nutrition  education 
cvirriculum  for  grades  K-3  is  provided  to  teachers  throughout  a 
State,  by  specifying  a  primary  outcome  of  changing  student "s 
health  status,  the  evaluator  sets  up  the  program  for  an  unfair  eval- 
uation. Changing  health  status  is  a  long-term  outcome  that  is 
affected  by  myriad  factors,  not  changed  by  a  single  intervention. 
More  realistic  short  term  results  for  a  classroom  nutrition  educa- 
tion program  could  include  increasing  student  knowledge  about 
nutrition  and  possibly  changing  attitudes  about  food  and  eating. 


The  NET  Coordinator  may  be  faced  with  a  dilemma  when  specify- 
ing program  outcomes.  Polio.-  makers  and  sometimes  program 
developers  can  have  different  expectations  of  the  outcomes  of 
nutrition  education  and  training  programs.  The  evaluator  of  a 
nutrition  education  program  or  project  must  balance  a  program 
practitioners'  reasonable  expectations  of  changes  in  knowledge  and 
attitude  toward  food  at  school,  with  policy  makers'  expectations  of 
more  distant  outcomes  of  changes  in  food-related  behaviors. 
St. Pierre  (1982)  discussed  proximity  of  an  outcome  and  three 
other  characteristics: 
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^   Fair  evaluation  in 
relation  to  proximity 
of  outcomes 

( Continued) 


First,  proximate  outcomes  (short-term  results)  tend  to  have 
low  policy  relevance  and  social  desirability  while  distant  out- 
comes (long-term  results)  are  high  on  this  dimension. 
Improvements  in  health  status  or  in  nutrition-related  behav- 
iors are  more  socially  desirable  outcomes  and  are  more  policy 
relevant  than  is  change  in  knowledge  about  nutrition. 
Second,  educational  programs  usually  have  a  larger  effect  on 
proximate  outcomes  than  on  distant  ones.  It  is  easier  for  a 
nutrition  education  program  to  teach  children  nutrition- 
related  knowledge  than  to  change  their  health  status.  Third, 
if  a  relationship  is  found  between  participation  in  a  program 
and  an  outcome,  it  is  easier  to  argue  that  the  relationship  is 
causal  when  the  outcome  is  proximate  rather  than  distant. 
Unfortunately,  limiting  evaluation  to  distant  outcomes  makes 
the  detection  of  effects  quite  difficult;  and  if  an  effect  is 
found,  attribution  to  the  treatment  is  almost  impossible. 


St. Pierre,  one  of  the  authors  of  the  1981  evaluation  of  the  NET 
Program  summarized  the  research  at  that  time,  "Little  research 
evidence  supports  a  clear  and  consistent  relationship  between 
increased  nutrition-related  knowledge  and  behavior  change.  Given 
the  range  of  factors  that  influence  a  child's  food-related  behavior, 
behavior  change  may  not  be  a  fair  outcome  for  in-school  nutrition 
education  programs  which,  in  general,  do  not  address  many  of  the 
out-of-school  factors  that  influence  children's  eating  patterns." 
Fourteen  years  later,  this  opinion  seems  to  still  hold  true  and  has 
important  implications  for  the  design  of  fair  evaluation  for  nutri- 
tion education  programs  and  projects. 


The  NET  Coordinator  should  address  these  four  important  decisions  discussed  above 
regarding  how  an  evaluation  plan  is  designed.  Decisions  regarding  how  to  approach  each 
issue  will  be  individual  to  each  evaluation  plan.  The  decision-making  process  and  the 
design  of  the  NET  Evaluation  Plan  will  be  enhanced  when  input  is  obtained  from  individ- 
uals involved  in  the  various  projects  and  activities  of  NET  and  from  individuals  who  rep- 
resent participant  groups.  The  Needs  Assessment  Guide  For  The  Nutrition  Education  and 
Training  Program  (1994,  page  38)  includes  a  discussion  of  the  advantages  of  involving  a 
task  force  in  the  needs  assessment  process.  If  the  State  agency  does  have  a  Needs 
Assessment  Task  Force  or  some  other  advisory  group  for  the  program,  it  is  desirable  to 
obtain  input  from  that  gtoup  regarding  the  evaluation  plan  as  well.  The  task  force  should 
be  provided  the  report  of  the  needs  assessment  and  their  input  should  be  sought  to  define 
program  priorities  and  to  develop  evaluation  plans.  The  task  force  can  also  provide  valu- 
able insight  on  practical  evaluation  measures  for  the  defined  program  objectives  and  on 
standards  to  assess  acceptability  or  measure  success  for  the  objectives. 
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Managing  Evaluation 

Developing  a  plan  for  evaluation  involves  the  same  steps  as  developing  a  plan  for  a  needs  assessment. 
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Step  1.  Develop  an  evaluation  plan  for  each  objective  in 


the  State  Plan. 
Include: 

^  Evaluation  questions  for  each  objective 
(program,  project,  or  performance 
objective), 

^  Eor  each  evaluation  question,  identifica- 
tion of  data  to  be  collected, 

^  For  each  evaluation  question,  identifica- 
tion of  standard  to  assess  acceptability, 

^  Eor  each  evaluation  question,  identifica- 
tion of  appropriate  data  collection 
method,  or  technique,  and 

^  A  work  schedule  with  assigned 
responsibilities. 

Step  2.  Determine  how  the  evaluation  plan  will  be  man- 
aged, either  in-house  or  through  a  contractor. 

Step  3.  Monitor  the  evaluation  plan. 

Step  4.  Analyze  and  interpret  results  of  the  evaluation. 

Step  5.  Use  the  evaluation  findings  in  management  deci- 
sion making,  determining  program  policies  and 
procedures,  and  for  program  accountability. 
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Step  1.  Develop  an  evaluation  plan  for  each  objective  in  the  State  Plan. 

The  State  Plan  objecrives  must  be  written  in  such  a  way  that  they  can  be  measured.  Then,  the 
NET  Coordinator  should  develop  an  evaluation  plan  for  each  objective  in  the  State  Plan.  This 
should  occur  during  the  developmental  phase  of  the  State  Plan. 


When  writing  objectives  for  a  specific  program  or  project,  the  same  process  should  be  used.  The 
program,  project,  or  performance  objectives,  define  what  should  be  evaluated. 

The  evaluation  plan  for  each  objective  should  include  the  following  information: 


^  Evaluation 

question(s)  for  each 
objective 

(program,  project,  or 
performance). 


Evaluation  questions  identify  the  specific  purpose(s)  of  the  evalua- 
tion, that  is,  what  you  want  to  know  from  the  evaluation.  The 
evaluation  questions  should  be  developed  during  the  planning 
phase  of  the  program,  not  after  the  program  has  begun,  or  has 
been  completed.  They  are  essentially  the  reasons  for  evaluation, 
and  as  such,  affect  the  baseline  data  that  will  be  collected  and  the 
data  collection  method(s)  chosen.  Some  evaluators  write  evaluation 
questions  as  evaluation  objectives  using  the  traditional  style  of 
objectives.  For  purposes  of  NET  evaluation,  evaluation  questions 
are  recommended  since  they  provide  a  clear  focus  for  the  subse- 
quent collection  of  evaluation  data  and  are  not  confused  with  State 
Plan  objectives. 


The  evaluation  questions  should  address  all  components  of  pro- 
gram or  project  planning  and  implementation.  The  questions 
should  be  written  to  obtain  a  multi-dimensional  view  of  what  is 
working  or  not  working  as  planned,  what  worked  or  did  not  work 
as  planned,  and  why.  This  brings  up  the  issue  of  types  of  evalua- 
tion and  evaluation  models.  An  explanation  of  one  of  the  most 
popular  evaluation  models  is  included  in  Chapter  3.  The  model 
includes  five  types  of  evaluation  which  provide  a  comprehensive 
approach  for  defining  evaluation  questions. 


Some  examples  of  evaluation  questions  of  various  types  are  shown 
below. 

^  How  did  the  situation  in  which  the  program  was  developed 
and  implemented  affect  the  program  implementation? 

^  Were  the  resources  for  the  program  adequate  for  program 
implementation? 

^  Was  the  program  delivered  as  it  was  planned? 

^  What  processes  during  program  implementation  were  successful? 

^  To  what  extent  were  program  objectives  met? 

^  What  changes  occurred  as  a  result  of  the  program? 
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^    For  each  evaluation 
question,  identify- 
data  to  be  collected. 


The  evaluation  questions  are  used  to  determine  the  t^-pes  of  data 
that  must  be  collected.  Some  examples  of  data  include:  attitudes, 
behavior,  opinions,  numbers,  program  costs,  productivity,  time, 
behavior,  qualit)-,  personal  reactions,  perceptions,  or  observations. 


Data  to  be  collected  for  each  evaluation  question  should  be 
described  in  the  evaluation  plan.  The  data  descriptors  should  meet 
the  following  criteria: 

1.  Be  meaningful  and  relevant  to  the  evaluation  question  (or 
evaluation  objective); 

2.  Be  available  and  easily  accessible; 

3.  Provide  value  to  justify'  the  cost  of  obtaining  the  data.  In 
other  words,  the  cost  of  the  evaluation  should  be  appropri- 
ate to  the  cost  of  the  program; 

4.  Be  obser\-able  and  measurable;  and 

5.  Be  timely. 

Data  collection  may  involve  an  entire  NET-targeted  population  or 
it  may  involve  a  sample.  When  quantitative  evaluation  methods 
are  used,  the  sampling  protocol  becomes  very  important.  Wlien 
qualitative  methods  are  used,  sampling  becomes  less  of  an  issue. 
There  are  a  variety  of  sampling  procedures.  A  description  of  prob- 
ability' sampling  is  included  in  Appendix  E.  This  Sampling 
Protocol  w^as  originally  prepared  for  use  by  NET  Coordinators  in 
sampling  for  Statewide  needs  assessments.  Additional  references 
on  sampling  are  listed  in  the  Selected  Bibliography  of  the 
Sampling  Protocol. 


^    For  each  evaluation 
question,  identifv'  a 
standard  to  assess 
acceptabilit\\ 


Standards  represent  the  pre-determined  yardsticks  that  will  be 
used  to  measure  success  or  to  determine  acceptability.  Standards 
for  evaluation  must  be 

^  reasonable, 

fi^  measurable,  and 

^  observable. 


They  may  be  described  with  numbers,  percentages,  or  behavioral 
descriptions  that  are  clear  to  everyone  involved.  As  part  of  the 
planning  process,  the  NET  Coordinator,  in  collaboration  with  oth- 
ers involved  in  the  program,  should  set  reasonable  standards  using 
his/her  expertise  in  the  subject  area. 


It  is  most  important  that  the  standards  be  defined  during  the 
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planning  stage  with  input  from  those  people  who  will  be  involved 
in  the  program  or  project.  For  example,  in  the  case  of  training 
programs,  the  NET  Coordinator,  other  State  staff  members 
involved  in  the  training,  representatives  of  the  target  groups,  and 
the  trainers  themselves  should  have  input  into  what  will  be  con- 
sidered "acceptable"  results  of  the  training. 

Consider  the  example  of  a  training  program  for  school  nutrition 
managers  on  using  production  records  to  forecast  amounts  to  pre- 
pare for  a  choice  school  lunch  program.  The  performance  objective 
might  be  stated  as, 

"The  manager  will  demonstrate  the  efficient  use  of  production 
records  for  forecasting  by  reducing  the  amount  of  leftovers  of  main 
dish  items  prepared  in  the  school  kitchen." 

The  evaluation  question  could  then  be,  "Can  the  manager  use  pro- 
duction records  for  forecasting  to  reduce  the  amount  of  leftovers  of 
main  dish  items  prepared  in  the  school  kitchen?"  The  data  for  this 
evaluation  question  could  be  obtained  at  the  end  of  the  training 
program  by  having  the  managers  complete  a  case  study  in  which 
they  demonstrate  use  of  production  records  for  forecasting  where 
they  reduce  the  potential  amount  of  leftovers.  The  completed  case 
study  could  be  included  in  a  portfolio  assembled  to  reveal  the 
manager's  proficiency  in  content  areas  addressed  in  the  training 
program.  The  standard,  which  was  pre-determined  when  the  eval- 
uation was  planned,  could  specify  that  85%  of  the  managers  in  the 
training  program  could  use  the  production  record  for  forecasting 
based  on  defined  criteria  in  the  case  study. 


^    For  each  evaluation 
question,  identify  an 
appropriate  data 
collection  method  or 
technique. 


The  evaluator  is  now  ready  to  identify  a  data  collection  method  or 
technique  (Phillips,  1991)-  When  selecting  data  collection  meth- 
ods and  techniques,  consider  the  following: 

1  ■  Whether  the  data  collection  method  can  be  used  to  attain  the 
data  needed  to  answer  the  evaluation  question. 
For  example,  when  data  are  needed  to  determine  the  extent 
to  which  training  program  participants  have  attained  specific 
skills  defined  by  performance  objectives,  the  data  collection 
method  must  provide  evidence  of  the  new  behavior.  A 
method  which  only  obtained  data  relating  to  participants' 
personal  reactions  to  the  training  would  not  answer  the  ques- 
tion, "can  they?" 
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^    For  each  evaluation 
question,  identify-  an 
appropriate  data 
collection  method  or 
technique. 

(  Continued) 


2.  How  the  evaluation  data  will  be  used. 

Knowing  the  purpose  of  evaluation  data  will  help  to  guide 
the  selection  of  the  method  used  to  obtain  it.  An  effective 
evaluation  gathers  facts.  The  object  is  to  determine  which 
facts  are  needed  to  answer  the  evaluation  question. 

0.  How  the  data  will  be  collected,  analyzed,  and  repiorted. 
The  evaluator  must  consider  how  the  data  will  be  collected, 
tabulated,  summarized,  analyzed,  and  reported.  The  time 
involved  and  the  expertise  needed  to  collect  the  data  will 
influence  the  data  collection  method  chosen.  When  external 
consultants  with  expertise  in  research  design  are  given 
responsibilin,-  for  evaluation,  a  more  sophisticated  and  in- 
depth  evaluation  method  can  be  used.  Consultants  generally 
work  on  a  contractual  basis  and  are  able  to  devote  more  time 
to  managing  a  comprehensive  evaluation  than  can  a  State 
agency  staff  member. 

4.  Who  will  use  the  information. 

The  target  audience  for  the  evaluation  summat}-  or  report 
should  be  considered  when  determining  the  methods  that 
will  be  used.  If  the  target  audience  is  policy  makers  at  the 
Stare  or  national  level,  different  evaliaation  questions  may  be 
defined  than  when  the  target  audience  is  instructional 
designers  for  a  training  program. 

The  Seeds  Assessment  Guide  For  The  Nutrition  Education  and 
Training  Program  (1994,  page  44)  includes  a  detailed  descrip- 
tion of  the  five  most  commonly  used  data  collection  methods 
for  needs  assessment  studies:  (1)  written  surveys,  (2)  inter- 
views, (3)  observ-ations,  (4)  data  reviews,  and  (5)  focus 
groups.  Since  these  same  methods  are  often  used  for  evalua- 
tion, the  evaluator  is  encouraged  to  review  each  technique 
(method)  before  deciding  which  is  the  most  appropriate  to 
obtain  data  for  answering  the  evaluation  question  while  con- 
forming to  the  constraints  of  the  program. 


In  addition  to  the  methods  already  mentioned,  the  descrip- 
tion of  three  data  collection  methods  useful  for  evaluation 
have  been  included  in  the  Appendices  of  this  Guide.  These 
include:  the  case  study,  the  participant  reaction  question- 
naire, and  the  portfolio  assessment. 
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Other  data  collection  methods  which  are  appropriate  for  use 
with  NET  evaluation  include  longitudinal  studies,  experi- 
mental design,  and  attitude  surveys.  Because  these  methods 
require  considerable  time  and  expertise  to  design  and  admin- 
ister, a  description  has  not  been  included  in  this  Guide. 
However,  most  research  methods  textbooks  include  a  detailed 
description  of  these  procedures.  The  Reference  section 
includes  a  listing  of  several  such  textbooks. 

Regardless  of  the  data  collection  method  to  be  used,  the  eval- 
uator  should  develop  an  instrument  that  is  consistent  with 
good  design  principles.  Refer  to  Appendix  A  for  Guidelines 
for  Developing  Evaluation  Instruments. 

^    A  work  schedule  Managing  program  evaluation  requires  a  systematic  approach 

with  assigned  based  on  a  management  plan.  The  evaluation  plan  should 

responsibilities.  include: 

1 .  A  work  schedule.  The  work  schedule  should  identify  the 
major  activities  or  tasks  required  to  answer  the  evaluation 
questions  or  implement  the  data  collection  method.  As  with 
any  aspect  of  program  management,  determining  the  time 
necessary  to  complete  an  activity  is  often,  at  best,  guesswork. 
However,  it  is  important  to  designate  beginning  and  ending 
times  for  each  activity  leading  to  a  completion  time-line  for 
each  part  of  the  evaluation. 

Two  of  the  most  popular  project  management  techniques  are 
network  models  and  Gantt  charts  (Martin  and  Trumbley, 
1987).  One  of  the  most  commonly  used  network  models  is 
the  Program  Evaluation  and  Review  Technique  (PERT).  The 
PERT  diagram  is  useful  in  the  planning  process  for  showing 
interdependent  and  dependent  tasks.  This  can  be  a  major  aid 
to  the  program  planner  by  helping  to  identify  which  tasks 
can  be  performed  simultaneously  and  which  tasks  must  be 
performed  consecutively.  The  program  planner  is  required  to 
think  backwards  from  the  desired  outcome,  identifying  the 
key  tasks  that  must  be  performed  to  achieve  the  evaluation 
objective.  A  flow  chart  is  then  planned  that  indicates  the 
path  or  route  for  performing  those  tasks  with  estimated 
time-lines  for  completion.  The  critical  path  is  identified  as 
the  path  which  links  the  tasks  that  must  be  performed  con- 
secutively, and  therefore,  take  the  longest  time  to  complete. 


^    For  each  evaluation 
question,  identify  an 
appropriate  data 
coUection  method  or 
technique. 

(Continued) 
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If  that  sequence  of  tasks  is  not  completed  on  time,  all  other 
aspects  of  the  project  are  negatively  affected. 


5^    A  work  schedule  The  Gantt  chart  is  based  on  a  horizontal  time  scale  that  is 

and  assigned  easily  referenced  for  quick-assessment  of  project  status  and  is 

responsibilities.  particularly  useful  for  project  monitoring.  To  develop  a 

(Continued)  Gantt  chart,  the  program  planner  lists  tasks  or  activities 

down  the  left  vertical  axis  of  a  graph  or  chart.  With  time 
intervals  on  the  horizontal  axis,  the  beginning  and  ending 
time  for  each  activin,*  is  projected.  The  time  inten'al  may  be 
days,  weeks,  or  months.  Time  frames  for  the  estimated  per- 
formance and  completion  of  specific  tasks  are  reflected  with 
the  use  of  solid  lines  under  the  date  or  time  interval  shown 
on  the  graph.  Actual  times  required  for  the  completion  of  the 
tasks  are  then  shown  with  a  broken  line  directly  under  the 
solid  project  time  line.  This  management  tool  is  easy  to 
develop,  is  useful  for  identifr^ing  overlapping  activities,  and 
serves  to  keep  a  project  on  schedule.  Gantt  charts  and  PERT 
charts  or  diagrams  are  described  in  the  Needs  Assessment  Guide 
For  The  Nutrition  Education  And  Training  Program  (1994,  pp. 
46-48). 

2.  Identification  of  personnel  assignments.  "WTien  evaluation  is 
conducted  "in-house,"  the  XET  Coordinator  may  be  respon- 
sible for  all  of  the  evaluation  tasks.  If  evaluation  is  part  of  a 
project  contract  or  is  a  separate  contract,  the  NET 
Coordinator  may  serve  as  the  project  director  or  the  project 
monitor.  In  either  case,  it  is  important  that  for  each  activity 
or  task,  the  responsibility  is  designated  for  someone.  Neither 
Gantt  charts  or  PERT  diagrams  include  personnel  assign- 
ments. However,  a  Gantt  chart  can  be  adapted  to  include  the 
person  responsible  for  each  task,  in  a  column  parallel  to  the 
task  column. 

An  example  of  a  systematic  plan  for  evaluation  of  NET  State 
Plan  objectives  is  included  in  Chapter  4. 
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Step  2.  Determine  how  the  evaluation  plan  will  be  managed,  in-house  or  through  a  con- 
tractor. For  contracted  projects,  provide  funding  for  the  contractor  to  include  an  evalua- 
tion as  part  of  the  project. 

When  a  project  proposal  includes  a  description  of  evaluation,  make  sure  the  work  schedule 
includes  the  tasks,  completion  dares,  and  personnel  responsibilities  for  all  aspects  of  the  evalua- 
tion plan.  This  will  provide  the  XET  Coordinator  a  basis  for  program  monitoring  throughout  the 
evaluation  process.  Section  6  of  the  SET  Seeds  Assessment  Guide  (1994,  pp.  56-63)  provides  guid- 
ance for  developing  requests  for  proposals  and  managing  external  projects. 

Step  3-  Monitor  the  evaluation  plan  as  part  of  monitoring  the  implementation  of  the 
NET  State  Plan. 

A  detailed  evaluation  plan  for  each  State  Plan  objective  enables  the  NET  Coordinator  to  monitor 
implementation.  A  Gantt  chart  provides  a  visible  tool  to  monitor  the  beginning  and  completion 
of  each  task  of  the  evalviation  plan. 

Step  4.  Analyze  and  interpret  results  of  the  evaluation. 

After  data  have  been  collected,  the  evaluator  should  follow  some  basic  guidelines  for  managing 
the  data.  (Phillips,  1991). 

^  Review  the  data  and  eliminate  data  collection  instruments  which  have  been  completed 
incorrectly  or  which  are  incomplete. 

^  Use  all  the  relevant  data.  Of  course,  any  program  manager  desires  positive  evaluations  and 
this  leads  to  a  built-in  bias.  Even  though  data  will  likely  contain  both  positive  and  negative 
information,  all  of  it  should  be  analyzed. 

^  Maintain  confidentialirv'  of  data.  The  same  security  used  for  maintaining  confidentiality 
during  data  collection  should  be  continued  during  analysis  and  reporting. 

^  Use  the  simplest  analysis  possible  to  answer  the  question.  In  most  cases,  descriptive  analysis 
can  be  used  to  answer  the  questions  posed  in  NET  evaluation.  Descriptive  treatment  of  data 
simply  describes  the  data.  The  results  can  be  presented  as  means,  medians,  modes,  and  fre- 
quenc}'  distributions.  This  analysis  provides  data  that  are  easy  to  display  in  various  kinds  of 
charts.  However,  it  cannot  be  used  to  identify  correlations  or  to  make  inferences  about  a 
large  population  from  a  small  sample. 

If  inferential  treatment  is  necessary,  the  appropriate  statistical  tests  should  be  used.  This  type 
of  statistical  treatment  of  data  describes  cause  and  effect  connections  or  relationships  among 
variables.  Use  caution  when  designing  the  analysis  so  that  the  resulting  data  do  not  become 
so  burdensome  and  detailed  that  the  reader  of  the  report  cannot  determine  the  key  results. 
Below  is  a  brief  summary  of  some  common  tests  lased  in  inferential  statistics. 
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Inferential  Treatment  of  Data 

Statistical  inference  comprises  those  methods  concerned  with  the  analysis  of 
a  subset  of  data  leading  to  predictions  or  inferences  about  the  entire  set  of 
data.  Some  of  the  more  common  tests  used  in  inferential  statistics  are  iden- 
tified below. 


Test 


t-test 


ANOVA 
(one-way) 

Pearson  Product  Moment 
Correlation  Coefficient 
ir) 

Phi  Coefficient 


Spearman  Rank 
Order  Coefficient 
(Rho) 


Definition 

To  compare  the  means  of  two  groups.  If  the  means  are 
far  enough  apart,  the  t-test  will  show  a  significant 
difference,  indicating  that  there  is  a  statistical  difference 
between  groups. 

To  simultaneously  compare  the  means  of  more  than  rw'o 
groups. 

Shows  whether  there  is  a  correlation  or  relationship 
between  two  sets  of  numbers,  r  does  not  imply  a 
cause-effect  relationship  among  variables. 

A  nonparametric  correlation  statistic  where  both  vari- 
ables are  divided  into  two  parts. 

A  correlation  statistic  where  the  data  are  ordinal  in  level, 
and  where  the  data  can  be  placed  in  rank  order.  The 
highest  score  is  ranked  "1"  and  so  on. 


Illustration  2.  Common  statistical  tests. 

Adapted  from:  American  Society  for  Training  and  Development.  (1991)-  Statistics  for  HRD  practice.  INFO-Line.  Author,  p. 10. 


The  data  analysis  provides  the  evaluator  with  the  basis  for  developing  the  evaluation  report.  The 
type  of  report  needed  will  vary  with  the  program  or  project,  and  the  evaluation  questions.  If  the 
evaluation  is  needed  only  by  the  NET  Coordinator  for  making  management  decisions,  it  may  not 
be  necessary  to  prepare  a  formal  report;  rather  the  Coordinator  may  draw  conclusions  from  the 
analysis  and  act  upon  them.  An  example  of  this  situation  would  be  the  formative  evaluation  of  a 
training  program  where  the  results  will  be  used  immediately  to  make  the  changes  needed  for  the 
next  scheduled  training  program. 


Nutrition  Education  &  Training  Program  U.S.  E>epartment  of  Agriculture 


In  most  situations,  the  evaluation  results  will  be  used  both  for  making  management  decisions 
and  for  program  accountabilir\-,  thus  a  formal  report  will  be  needed.  Evaluation  should  be 
designed  for  each  program  objective  included  in  the  State  Plan,  and  the  results  should  be  provid- 
ed with  the  subsequent  year's  State  Plan.  When  preparing  an  evaluation  report,  use  tables  and 
graphs  to  display  the  data,  limiting  narrative  to  a  description  of  the  visual  displays.  More  infor- 
mation on  how  to  write  a  report  to  present  the  findings  can  be  reviewed  in  the  SET  Seeds 
Assessment  Guide  (1994,  pages  52-53). 

For  evaluation  to  be  truly  meaningful,  a  review  of  the  analysis  should  include  time  for  reflection 
in  an  attempt  to  determine  the  real  meaning  of  the  results.  Reflection  upon  the  evaluation  find- 
ings along  with  personal  obser\'ations  and  impressions,  can  provide  the  evaluator  with  valuable 
insight  to  answer  the  evaluation  questions  (Steele,  1991)- 

Evaluation  becomes  most  valuable  when  it  helps  the  program  manager  think  creatively  and  think 
ahead.  Evaluation  by  its  ver\-  nature  deals  with  the  recent  past.  The  program  manager  must  ana- 
lyze current  trends,  changes  in  the  program,  and  changes  in  the  environment,  along  with  the 
evaluation  results  to  move  the  program  forward  to  fit  the  future.  Having  examined  the  analysis 
with  a  creative  and  critical  eye,  the  report  can  be  developed. 

Step  5 .  Use  the  evaluation  findings  for  management  decision  making,  for  determining 
program  policies  and  procedures,  and  for  program  accountabilit\'. 

The  purpose  of  evaluation  is  to  provide  data  to  continuously  improve  the  NET  Program  and  jus- 
tify its  funding.  Thus,  the  gathering  of  evaluation  data  and  analyses  will  be  of  no  value  unless  the 
findings  are  used  for  continuous  program  improvement. 

Summary- 
Developing  and  managing  evaluation  for  the  NET  Program  involves  a  step-by-step  process.  The 
process  includes  the  development  of  an  evaluation  plan,  management  of  the  plan,  monitoring  the 
plan  as  part  of  the  NET  State  Plan,  analyzing  and  interpreting  the  results  of  the  evaluation,  and 
using  the  e^'aluation  findings  for  program  management.  The  evaluation  plan  should  include  eval- 
uation questions  for  each  State  Plan  objective.  For  each  evaluation  question,  the  NET 
Coordinator  should  identify:  the  data  to  be  collected,  a  standard  to  assess  acceptability,  a  proce- 
dure including  an  appropriate  data  collection  method,  and  a  w-ork  schedule  with  assigned  person- 
nel responsibilities.  Evaluation  closes  the  loop  in  the  Assessment-Planning -Budgeting- 
Implementation-Evaluation  Process  for  managing  a  State  NET,  a  project,  or  program. 
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Chapter  III 
THE  CIPPI  Model 

4 

/or  Evaluation 


Characteristics  of  an  Evaluation  Model 

A  State  NET  Program  is  a  dynamic  program  continually  changing  to  meet 
the  identified  needs  of  the  targeted  populations.  Thus,  NET  is  essentially 
developmental  in  nature.  A  comprehensive  evaluation  model  is  needed  to 
examine  the  variety  of  projects  and  activities  that  are  part  of  NET.  An  evalu- 
ation model  appropriate  for  NET  should: 

^    recognize  the  dynamic  nature  of  the  Program; 

V    integrate  both  qualitative  and  quantitative  methods  of  analysis; 

<7    address  the  interdependence  ber?\'een  contextual  preconditions, 

program  deliver}-  attributes,  and  program  outcomes  (Edwards, 

1986);  and 

^  provide  the  State  NET  Coordinator  with  the  evaluation  informa- 
tion necessary  to  make  management  decisions  and  conform  to  the 
regulatory  requirements  of  the  NET  Program. 

There  are  a  variet}'  of  different  models  for  evaluation.  The  model  presented 
in  this  Guide  was  selected  because  it  addresses  basic  elements  appropriate  to 
the  evaluation  of  programs  which  are  developmental  in  nature.  The  CIPPI 
Model  for  evaluation  is  a  multi-dimensional  approach  for  the  collection  and 
assessment  of  data  needed  for  making  effective  and  cost-efficient  program 
management  decisions.  This  evaluation  model  precludes  the  evaluator  from 
taking  a  myopic  view  of  the  scope  of  evaluation  or  a  piecemeal  approach  to 
evaluation  design.  The  CIPPI  Model  is  one  of  the  most  commonly  used 
frameworks  in  the  design  of  a  comprehensive  evaluation  plan.  The  original 
model  for  CIPP  evaluation  w^as  based  on  the  work  of  Stiofflebeam  (19""!)  and 
was  further  refined  by  Trapnell  (1984)  to  include  impact  evaluation,  adding 
the  I,  thus  becoming  the  CIPPI  Model.  This  evaluation  model  provides  both 
proactive  information  support  for  management  decision  making  and  retroac- 
tive information  support  for  accountabilit}'.  Thus,  it  is  particularly  appropri- 
ate for  NET  evaluation. 
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The  CIPPI  Model 

The  CIPPI  Model  includes  five  major  types  of  evaluation. 
C  -  Context  Evaluation 
/    -  Input  Evaluation 
P  -  Process  Evaluation 
P  -  Product  Evaluation 
/   -  Impact  Evaluation 

This  multi-dimensional  approach  to  evaluation  provides  a  way  for  the  NET  Coordinator  to  exam- 
ine NET  as  a  total  phenomenon,  rather  than  a  series  of  isolated  projects  and  activities.  Each  type 
of  evaluation  provides  a  framework  for  determining  evaluation  questions.  Review  an  example  of  a 
NET  Evaluation  Plan,  shown  in  Chapter  4,  to  see  examples  of  questions  for  each  type  of  evalua- 
tion. 

The  following  is  a  brief  description  of  each  type  of  evaluation  comprising  the  CIPPI  Model. 
Chapters  5,  6,  7,  8,  and  9  provide  a  detailed  description  of  each  type  of  evaluation. 

®  CONTEXT  EVALUATION  (Chapter  5) 

This  type  of  evaluation  focuses  on  the  overall  social,  political,  economic,  and  envi- 
ronmental setting  in  which  NET  is  planned  and  implemented.  It  can  be  consid- 
ered a  situation  evaluation.  It  addresses  the  general  context  in  which  the  Program 
is  implemented  and  how  this  context  affects  the  achievement  of  the  Program's 
goals  and  objectives.  Success  or  the  perceived  lack  of  effectiveness  of  a  State  NET 
Program  may  lie  in  the  context  in  which  that  program  operates  rather  than  in  the 
design  or  delivery  of  the  program  itself.  Data  obtained  for  context  evaluation  may 
come  from  needs  assessment.  It  describes  problems,  organizational  climate,  barri- 
ers, opportunities,  and  value  orientations.  This  type  of  evaluation  is  generally  con- 
ducted with  administrative  or  management  personnel  in  the  State  agency,  school 
districts,  sponsoring  agencies,  or  cooperating  organizations. 

INPUT  EVALUATION  (Chapter  6) 

Input  evaluation  assesses  the  appropriateness  and  adequacy  of  all  human,  fiscal, 
and  physical  resources  that  go  into  the  design  and  implementation  of  a  given  pro- 
gram or  initiative.  Included  are  such  resources  or  inputs  as  equipment,  facilities, 
materials,  supplies,  curriculum  guides,  entry-level  ability  of  students,  and  the 
competency  level  of  the  instructional  staff  providing  the  education  or  training 
program.  Input  evaluation  is  usually  conducted  with  the  individuals  who  are 
involved  in  planning  and  implementing  a  program.  This  might  include  program 
administrative  or  management  personnel  in  the  State  agency,  school  districts,  or 
sponsoring  agencies;  or  trainers  and  others  involved  in  implementing  a  program 
or  project  at  the  local  level.  Program  participants  could  also  be  targeted  for  input 
evaluation. 
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In  a  training  program,  input  evaluation  would  provide  information  about  the  ade- 
quacy and  appropriateness  of  the  training  materials.  Included  would  be  the  iden- 
tification of  performance  objectives,  the  design  of  the  instructional  activities,  the 
planned  sequence  of  the  instructional  activities,  and  the  types  of  training  materi- 
als to  be  used  by  both  the  instructor  and  panicipant  (or  student).  Furthermore,  it 
would  address  the  adequacv'  of  the  equipment  and  facilities,  the  entry-level  behav- 
iors of  the  participants,  and  the  competency  level  of  the  instructor. 

^  PROCESS  EVALUATION  (Chapter  7) 

Process  evaluation  provides  the  NET  Coordinator  information  about  the  extent  to 
which  the  program  is  implemented  as  it  was  designed;  whether  resources  are  used 
appropriately  during  the  implementation  phase  of  the  program,  project,  or  initia- 
tive; and,  if  the  students  are  meeting  interim  performance  objectives.  This  rype  of 
evaluation  addresses  the  guiding  management  principles  of  "plan  your  work,"  and 
"work  your  plan."  It  provides  information  on  whether  and 'or  how  well  the  pro- 
gram or  project  plan  was  implemented.  Formative  evaluation,  which  occurs  dur- 
ing the  implementation  phase  of  a  program  or  project,  is  especially  critical  in 
process  evaluation,  as  it  can  provide  an  array  of  data  for  making  changes  during 
the  implementation  of  a  program  or  project.  Individuals  who  are  usually  the  audi- 
ence for  this  t}^e  of  evaluation  include  program  administrative  or  management 
personnel  in  the  State  agency,  school  districts,  and  sponsoring  agencies;  trainers 
and  others  involved  in  implementing  a  program  or  project  at  the  local  level;  and 
program  participants. 

Process  evaluation,  which  occurs  during  implementation  of  a  training  program, 
can  provide  feedback  regarding: 

^  the  effectiveness  of  the  instructor  in  utilizing  the  resources, 
^  interim  performance  gains  of  the  participants  (or  students),  and 
^  implementation  of  the  program  as  it  was  designed. 

Using  the  evaluation  data,  the  NET  Coordinator  can  make  important  changes  in 
the  program  during  the  implementation  process. 

B  PRODUCT  EVALUATION  (Chapter  8) 

This  is  the  type  of  evaluation  with  which  most  people  are  familiar.  It  fooises  on 
participant  (or  student)  outcomes,  ability  levels,  and  performance  after  the  com- 
pletion of  a  program.  It  may  be  referred  to  as  the  "proof  of  the  pudding"  because 
it  determines  the  extent  to  which  students  attain  the  performance  objectives 
included  in  the  curriculum  guide  or  training  workbook  and  whether  the  students 
can  apply  those  skills  in  a  variety  of  settings.  Product  evaluation,  thus,  closes  the 
loop  in  curriculum  and  instructional  design.  Product  evaluation  audiences  are  pri- 
marily program  participants  who  may  include  adult  learners  or  K-12  students. 
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Summative  evaluation  is  the  most  common  design  in  product  evaluation,  as  the 
evaluation  takes  place  after  students  have  completed  a  course  or  program. 
Although  product  evaluation  has  been  the  focus  of  the  majorirv-  of  evaluation 
efforts  over  the  years,  it  represents  only  the  tip  of  the  iceberg  in  the  design  of  a 
comprehensive  evaluation  effon. 

IMPACT  E\:\LUATIOX  (Chapter  9) 

Impact  evaluation  focuses  on  the  cximulative  difference  NET  makes  on  the  target 
groups  it  is  mtended  to  serve,  and  on  the  problems  or  needs  it  is  intended  to 
address.  Thus,  impact  evaluation  may  be  considered  "the  bottom  line"  evaluation. 
This  rv'pe  of  evaluation,  more  than  any  other,  provides  the  justification  for  bud- 
getan.'  requests,  especially  for  budget  increases,  as  it  documents  the  ultimate 
effectiveness  of  the  NET  Program.  Impact  evaluation  provides  a  picture  of  long- 
term  program  impact  or  outcome  rather  than  the  difference  one  training  program 
or  one  nutrition  education  project  made. 


The  CIPPI  ^^odel  for  evaluation  provides  the  NET  Coordinator  a  comprehensive  approach  to  col- 
lecting and  analyzing  data  about  NET  throughout  the  planning  and  implementation  process.  The 

various  tv'pes  of  evaluation,  taken  together,  can  provide  a  overall  picture  of  the  program  and  its 
major  projects  that  will  support  sound  management  decisions. 

Summary- 

The  CIPPI  Model  can  be  used  to  design  the  evaluation  for  large  programs  such  as  a  State  NET 
Program,  large  projects  such  as  a  State  training  program,  or  small  projects  such  as  a  mini-grant 
to  a  school  district  or  Child  and  Adult  Care  Food  Program  (CACFP)  center  sponsor.  It  provides  a 
multi-dimensional  approach  to  evaluation  which  enables  the  NET  Coordinator  to  design  a  mosaic 
with  each  piece  of  collected  data  assembled  into  a  whole  that  is  more  meaningful  to  program 
improvement  than  the  individual  evalviation  parts. 
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Chapter  IV 
Application  of  the  CIPPI  /Model 
NET  Evaluation  Plan 

The  CIPPI  Model  for  evaluation  addresses  the  program  performance,  rvpical  of  most  all  evalua- 
tion models,  as  well  as  the  processes  involved  in  program  delivery.  By  examining  a  program  using 
this  multi-dimensional  approach,  the  program  planner  can  achieve  a  better  understanding  of 
what  makes  a  program  successful  or  what  needs  to  be  improved  to  achieve  success.  Using  the  var- 
ious vrpts  of  evaluation  in  the  CIPPI  Model,  the  NET  Coordinator  should  develop  an  evaluation 
plan  for  each  NET  State  Plan  objective. 

Because  context  evaluation  provides  an  analysis  of  the  situation  in  which  a  program  operates,  in 
some  cases  it  may  be  more  appropriate  to  gather  and  analyze  context  evaluation  data  regarding 
the  whole  NET  Program  rather  than  specific  objectives  of  the  State  Plan.  This  will  provide  the 
NET  Coordinator  insight  into  the  social,  political,  economic,  and  physical  aspects  of  the  environ- 
ment which  can  affect  the  achievement  of  the  program's  goals  and  objectives.  For  comprehensive 
State-initiatives,  where  there  are  several  State  Plan  objectives  with  one  focus,  the  NET 
Coordinator  may  wish  to  obtain  context  evaluation  data  in  order  to  achieve  an  understanding  of 
sinaational  factors  which  contribute  to  overall  program  success  or  hinder  program  implementa- 
tion. For  such  an  initiative,  context  evaluation  data  should  be  obtained  from  State-level  adminis- 
trators and  program  personnel;  from  individuals  in  targeted  school  districts,  schools,  centers;  and 
from  other  targeted  agencies.  An  example  could  be  the  implementation  of  the  Dietary 
Guidelines,  where  there  are  many  different  programs  and  projects  with  the  same  foaas. 

Impact  evaluation  is  an  examination  of  cumulative  differences  resulting  ffom  program  initiatives 
over  the  long  term.  As  such,  this  type  of  evaluation  is  not  conducted  for  short-term  training  pro- 
grams or  for  individual  nutrition  education  interventions.  Rather,  it  is  appropriate  to  examine  the 
impact  of  all  aspects  of  training  over  a  period  of  time  and  is  appropriate  to  examine  the  difference 
made  by  an  extensive  training  program,  such  as  one  which  provides  statewide  training  for  all 
school  nutrition  managers  in  a  30-hour  course  with  accompanying  school  district  follow-up.  This 
type  of  training  can  be  expected  to  impact  on  the  delivery  of  services  in  school  nutrition  depart- 
ments. A  one-time  training  program  of  six  hours  for  school  nutrition  managers  may  affect  atti- 
tudes and  increase  knowledge  and  skills,  but  is  less  likely  to  have  a  significant  bottom-line 
impact  on  the  deliver}'  of  food  ser\'ices.  Impact  evaluation  is  meful  for  examining  the  impact  of 
several  different  programs  or  projects  with  the  same  focus,  for  example,  implementation  of  the 
Dietax)'  Guidelines. 

The  evaluation  plan  included  in  this  chapter  illustrates  how  the  NET  Coordinator  can  plan  evalu- 
ation for  State  Plan  objectives.  It  is  provided  as  an  example  of  the  application  of  the  CIPPI  Model 
and  is  not  intended  as  a  requirement,  or  as  the  one  recommended  way  to  use  the  Model.  For  pur- 
poses of  illustration,  three  different  NET  Strategic  Plan  goals  have  been  used  in  this  scenario 
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together  with  a  State  objective  for  each  goal.  For  each  State  objective,  one  or  more  implementa- 
tion activities  have  been  included.  Generally,  a  State  would  include  more  than  one  State  Program 
objective  tor  each  goal.  The  CIPPI  Model  is  applicable  to  projects  and  other  initiatives  as  well. 


XET  Strategic  Plan  -  Goal  1:  Increase  to  at  least  75%  the  proportion  of  the  Nation's 
schools  that  provide  nutrition  education  as  part  of  health  education,  as  part  of  an  interdis- 
ciplinar\-  nutrition  education  curriculum,  or  through  other  means,  to  children  and  parents 
by  the  year  2000,  as  documented  by  data  submitted  to  FNS  or  by  other  sampling  meth- 


State  Plan  Objective  1:  To  reach  309r  of  the  K-4  students  with  nutrition 
education  through  the  use  of  the  State  Wellness  Curriculum,  nutrition 
education  section. 

XET  Strategic  Plan  -  Goal  3:  Increase  to  at  least  50%  the  proportion  of  food  service  per- 
sonnel providers  in  Child  Nutrition  Programs  who  have  received  nutrition  information 
and'or  training  (including  food  service  training)  by  the  year  2000,  as  doctamented  by  data 
submitted  to  FNS  or  by  other  sampling  methods. 

State  Plan  Objective  2:  Develop  and  implement  a  train-the-trainer  pro- 
gram to  present  a  10-hour  Dietar\'  Guidelines  workshop  to  at  least  400 


NET  Strategic  Plan  -  Goal  2:  Increase  to  at  least  50%  the  proportion  of  child  care  and 
summer  program  facilities/sites  that  provide  nutrition  informatioa'education  to  children, 


ods. 


managers. 


caregivers,  or  parents  by  the  year  2000,  as  documented  by  data  stibmitted  to  FNS  or  by 
other  sampling  methods. 


State  Plan  Objective  3:  To  reach  at  least  1000  children  in  child  care  cen- 
ters with  nutrition  education  learning  experiences  consistent  with  the 
principles  of  the  Food  Guide  Pyramid. 
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Chapter  V 
Explanation  of  Context  Evaluation 


When  to  Use  It 

^  To  examine  the  climate  and  cul- 
ture of  the  environment  in  which 
XET  operates  at  the  State  level; 

and 

^  To  examine  the  sitxiation  in  which 
major  projects  or  programs  oper- 
ate at  the  school  district,  school, 
or  sponsoring  agenc}"  level. 

How  to  Design  It 

Lends  itself  to  qualitative  methods 

including: 

^  Direct  observation. 

Ethnography, 
%  Interviews, 
^  Focus  groups,  and 
^  Cognitive  response  techniques. 

Who  is  Targeted? 
Administrative  and  management  per- 
sonnel in: 
^  State  agencies, 
^  School  districts, 
%  Sponsoring  agencies,  and 
^  Cooperating  organizations. 


Introduction 

Context  evaluation  focuises  on  the  overall  social, 
political,  economic,  and  physical  environment  in 
which  the  nutrition  education  and  training  program 
is  planned  and  implemented;  the  goals  and  objectives 
of  the  program;  and  how  the  goals  are  perceived  by 
different  individuals.  It  answers  the  primary  ques- 
tion, "In  what  way(s)  did  the  context  in  which  the 
project  or  program  was  conducted  afifect  the  achieve- 
ment of  the  goals  and  objectives?'"  Extensive  research 
has  shown  that  contextual  factors  such  as  organiza- 
tional climate  and  culture  significantly  afreet  imple- 
mentation of  any  new  approach.  Unfortunately,  these 
important  considerations  are  generally  ignored  in 
program  evaluations,  particularly  in  evaluations  of 
staff  development  (Guskey  and  Sparks,  1991). 

Context  evaluation  focuses  on  the  macro-environment 
of  the  program,  which,  in  many  ways,  establishes  its 
scope  of  influence  and  public  image.  It  addresses  how 
external  factors  shape  and  mold  the  purpose  and 
comprehensiveness  of  the  State's  Child  Nutrition 
Program,  and,  in  turn,  help  to  shape  the  perspective 
of  various  individuals  relative  to  the  importance  and 
purpose  of  the  program.  Those  perceptions  become  a 
major  contributing  factor  to  defining  the  parameters, 
goals,  and  objectives  of  a  given  program. 


Why  Address  this  T\-pe  of  Evaluation? 

Context  evaluation  helps  a  NET  Coordinator  understand  the  situation  (context)  in  which  the 
Program  is  to  bring  about  change.  One  important  area  of  study  in  context  evaluation  may  be 
determining  how  State,  district,  and  school  administrators  view  nutrition  education  and  training, 
and  the  degree  to  which  they  display  support  and  commitment  for  the  program.  If,  for  example, 
extensive  staff  development  programs  have  been  conducted  by  the  State  agenc\'  to  develop  collab- 
orative planning  efforts  among  faculn.-  and  school  nutrition  personnel,  but  the  school  principals 
fail  to  provide  joint  planning  time  in  the  school  schedule,  the  probability  of  the  staff  develop- 
ment program  enhancing  collaborative  planning  efforts  is  restricted  regardless  of  how  good  the 
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program  might  be.  This  is  an  example  of  how  a  State  training  program  can  be  adversely  affected 
by  the  context  in  which  implementation  ocairs  at  the  local  level. 

In  this  example,  if  the  evaluation  only  fooised  on  the  resources  (input),  the  implementation 
process,  or  participant  outcomes  of  the  staff  development  program,  the  root  cause  of  the  problem 
may  go  undetected.  If,  on  the  other  hand,  this  t\-pe  of  evaluation  data  had  been  provided,  the 
program  planners  could,  in  the  next  planning  cycle,  identifH-  and  offer  strategies  to  address  the 
school  administrators'  perspective  prior  to  offering  the  staff  development  program. 

When  to  Use  Context  Evaluation 

Context  evaluation  provides  the  NET  Coordinator  information  about  the  climate  and  culture  of 
the  environment  in  which  a  program  operates.  This  is  important  in  evaluation  of  the  State  XET 
Program  as  a  whole  and  it  is  important  in  evaluation  of  major  developmental  projects.  Context 
evaluation  for  a  training  program  which  focuses  on  the  environment  of  an  organization  such  as  a 
school,  a  school  district,  or  a  State  training  system,  provides  decision  makers  with  information 
regarding  how  that  environment  influenced  the  success  or  failure  of  the  training  program.  For 
example,  the  success  of  a  State  staff  development  (training)  deliver}-  system  may  be  partly  attrib- 
utable to  appropriate  planning  that  provided  for  a  support  network  of  school  nutrition  adminis- 
trators and  school  nutrition  trainers  throughout  the  State.  This  State  network  for  staff  develop- 
ment can  provide  for  an  on-going  delivery  system  for  all  training  programs  with  continuing  sup- 
port to  the  trainers  and  participants  from  the  State  and  school  district  levels.  When  the  focus  of 
context  evaluation  is  on  the  overall  NET  Program,  an  example  of  a  contextiaal  evaluation  ques- 
tion which  may  be  addressed  is,  "What  is  the  relationship  of  the  organizational  location  of  the 
XET  Program  within  State  government  (i.e.,  as  part  of  the  Child  Nutrition  Program  staff,  as  part 
of  the  curriculum  staff,  or  as  part  of  the  Department  of  Public  Health)  to  the  perceived  impor- 
tance and  the  administrative  support  of  the  Program  by  State-level  administrators?" 

Thus,  it  is  important  that  context  evaluation  be  addressed  in  a  comprehensive  manner  since  it  can 
provide  strategic  clues  for  projecting  the  ultimate  success  of  a  program  or  examining  why  a  pro- 
gram or  project  was  or  w^as  not  successful.  When  contextual  questions  are  addressed  in  the  initial 
stages  of  the  program  design,  they  may  help  to  prevent  failure.  If,  on  the  other  hand,  these  types  of 
data  are  collected  summatively,  the  findings  can  provide  direction  for  future  program  planning. 

How  to  Design  Context  Evaluation 

Context  evaluation  is  not  an  exact  science  and  lends  itseff  more  appropriately  to  qualitative 
research.  The  focus  can  be  on  the  subtleties  and  nuances  contributing  to  the  public  image  of  the 
program  as  well  as  on  the  complexities  which  fashion  the  scope  of  a  program.  Direct  observation, 
ethnography,  interviews,  focus  groups,  and  cognitive  response  techniques  are  commonly  used 
methods  for  collecting  data  using  a  qualitative  evaluation  design.  Guidelines  and  techniques  for 
using  observation,  interviewing,  and  focus  group  methods  are  described  in  detail  in  the  Appendix 
of  the  Seeds  Assessment  Guide  For  The  Sutrition  Education  And  Training  Program  (1994,  pp.  ~l-88). 
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Ethnography  is  a  type  of  observation  methodology  that  focuses  on  observing  the  participants  in 
the  context  of  their  daily  lives  over  an  extended  period  of  time.  Kolasa  (1981)  suggests  that 
ethnography  has  all  the  advantages  of  direct  obsen-ation,  in  addition  to  producing  a  "deeper, 
richer  understanding  of  context  and  meaning."  Although  this  type  of  evaluation  can  be  exp>ensive 
when  obser\-ers  are  paid  evaluators,  it  could  be  the  most  appropriate  method  for  certain  kinds  of 
nutrition  education  and  training  projects. 

For  example,  ethnography  could  be  used  as  an  evaluation  method  in  a  pilot-project  school  receiv- 
ing NET  funds  to  develop  a  model  for  school  nutrition  personnel  to  work  effectively  with  special 
education  teachers  and  parents  to  meet  the  special  dietan,-  needs  of  those  students  with  diet  pre- 
scriptions. For  this  kind  of  pilot  project,  where  one  of  the  purp>oses  would  be  to  replicate  it  in 
other  schools,  it  would  be  important  to  describe  the  context  in  which  the  teamwork  developed.  A 
trained  observer  could  be  funded  to  observe  the  interactions  of  teachers,  school  nutrition  person- 
nel, and  parents  to  identify*  those  behaviors  that  promoted  teamwork,  problem  solving,  and  an 
atmosphere  of  support  and  trust.  The  usefulness  of  context  evaluation  in  this  example  is  obvious. 
The  project  could  not  be  replicated  in  another  school,  no  matter  how  good  the  model,  without 
having  or  developing  the  sajne  kind  of  organizational  climate  in  the  school. 

Another  method  appropriate  for  context  evaluation  is  the  cognitive  response  method.  This 
method  requires  respondents  to  "verbalize  the  thoughts,  feelings,  and  ideas  that  come  to  mind 
while  examining  or  reading  a  message"  (Achterberg,  1988).  This  method  can  be  especially  help- 
ful in  tr\-ing  to  determine  the  attitudes  of  different  groups  relative  to  the  importance  of  nutrition 
education  and  training  and  or  to  the  goals  and  objectives  of  particular  programs.  For  example, 
this  method  might  be  appropriate  for  use  in  obtaining  context  evaluation  data  from  school 
administrators  regarding  the  issue  of  allowing  the  sale  of  snack  foods  in  the  school  and  where  the 
proceeds  should  be  used.  A  series  of  pictures  could  be  shown  to  a  sample  of  principals  to  obtain 
their  reaction.  Pictures  could  include  soft  drink  machines;  vending  machines  with  mostly  high- 
fat,  salty  snack  foods;  vending  machines  with  mostly  fresh  fruit,  reduced  fat,  and  reduced  salt 
snacks;  a  snack  bar  run  by  the  athletic  department;  and  sale  of  candy  by  school  clubs.  School 
administrators'  responses  to  these  pictures  could  provide  the  NET  Program  important  contextual 
evaluation  data  for  use  in  planning  nutrition  education  projects. 

In  the  qualitative  methods  described,  randomness  in  sample  selection  is  not  a  concern,  and  may, 
as  Achterberg  suggests,  be  counterproductive  to  what  is  needed  (Ibid.,  1988).  Sampling  for  these 
qualitative  evaluation  methods  is  focused  on  inclusiveness  and  on  selecting  individuals  w^ho 
exhibit  a  full  range  of  characteristics  of  the  population.  Since  standard  statistical  tests  are  rarely 
used,  sample  size  is  not  as  important  as  the  inclusiveness  of  characteristics,  thus,  sample  size  may 
be  limited  to  a  relatively  small  group  of  10  to  20  people. 

Once  the  data  have  been  collected,  they  are  aggregated  into  broad,  topical  categories.  Next,  a 
theme  analysis  is  conducted  by  studying  the  data  to  identify-  the  presence  of  any  underlying 
themes.  These  analyses  are  then  used  to  construct  a  theoretical  framework  which  attempts  to  syn- 
thesize the  data  into  a  meaningful  whole.  This  produces  what  is  referred  to  as  a  "grounding  theo- 
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ry  "  which  helps  to  ensure  the  validity  of  the  data  and  provides  a  framework  for  describing  the 
findings  (Glaser  and  Strauss,  196"').  Consider  the  previous  example  of  the  cognitive  response 
method  used  to  obtain  contextual  evaluation  data  from  school  administrators  regarding  their  atti- 
tude toward  the  sale  of  snack  foods  in  their  school.  Data  gathered  from  principals  throughout  the 
State  would  be  aggregated  into  broad  categories  such  as:  primar}-  concern  with  nutritional 
integrit)-  in  the  school,  primary-  concern  with  meeting  student  expectations,  or  primary'  concern 
with  obtaining  extra  funds  to  pay  for  school  needs.  The  three  underlying  themes  then  could  be 
described  as,  "nutrition,  student  requests,  or  money  for  school."  This  type  of  context  evaluation 
would  be  useful  to  State  planners  as  well  as  school  district  nutrition  directors  for  designing  train- 
ing programs  for  school  administrators  and  for  preparing  to  meet  the  challenge  of  ensuring  nutri- 
tion integrity-  of  school  food  service  programs. 

Another  type  of  qualitative  research  design  is  the  data  review  method.  If  the  evaluator  is  interest- 
ed in  determining  the  demographic  profile  of  the  State  or  of  a  given  region,  with  a  particular 
focus  on  the  population's  societal  and  political  orientation,  the  review  of  current  demographic 
studies  and  statistics  may  prove  quite  valuable.  Information  on  this  design  can  be  found  in  the 
Appendix  of  the  SET  Seeds  Assessment  Guide  (1994). 

Summary' 

Context  evaluation  involves  the  examination  of  the  situation  in  which  a  program  or  project  oper- 
ates. It  lends  itself  to  qualitative  evaluation  methods  and  generally  targets  program  administra- 
tors in  State  agencies,  school  districts,  sponsoring  agencies,  and  cooperating  organizations. 
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JJ\^3\^1\  Chapter  VI 

[?r\fey\^y      Explanation  of  Input  Evaluation 


When  to  Use  It 

As  part  of  needs  assessment  to 

examine  resources  available  and 

competenc}'  of  instructional  staff; 

During  program  implementation 

(formative),  to  examine  adequacy' 

of  resources,  entn,-  level  abilit}'  of 

students,  and  competency*  level  of 

instructors;  and 

Following  program  implementa- 

tion (summative),  to  examine 

adequacy  of  resources,  entry  level 

abilit}'  of  students,  and  compe- 

tency level  of  instructors. 

How  to  Design  It 

Qualitative  methods  include: 

Direct  observation, 

Ethnography, 

Interviews, 

Focus  groups,  and 

Expert  judgment. 

Quantitative  methods  include: 

Survey  questionnaires, 

Stnictured  interviews,  and 

Experimental  design. 

Who  is  Targeted? 

Program  administrators  and  man- 

agement personnel; 

Individuals  involved  in  imple- 

mentation, e.g.  trainers  in  a  staff 

development  project;  and 

Program  participants. 

Introduction 

Input  evaluation  focuses  on  the  appropriateness  and  ade- 
quacy of  all  human,  fiscal,  and  physical  resources  that  go 
into  the  design  and  provision  of  a  given  program  or  ini- 
tiative. Included  are  such  resources  or  input  as  equif)- 
ment,  facihties,  materials,  supphes,  curriculum  guides, 
entry  level  abilit}-  of  students,  and  the  competency-  level 
of  the  instruCTional  staff  providing  the  education  or 
training  program.  In  other  words,  it  focuses  on  e\'ery- 
thing  that  goes  into  the  makeup  of  a  given  program  or 
projea. 

Why  Address  this  T\pe  of  Evaluation? 
Input  ev-aluation  can  make  a  major  contribution  to  deci- 
sion making  by  providing  information  useful  for  exam- 
ining alternative  strategies,  designs,  and  resource  needs 
for  the  NET  Program  or  its  projects.  For  example,  input 
evaluation  could  provide  important  data  regarding  the 
use  of  a  State  nutrition  education  curriculum  gxaide 
which  calls  for  extensive  use  of  computer  software  in 
classroom  settings  where  the  instructors  lack  computer 
skills,  or  the  classrooms  do  not  have  adequate  computers 
for  student  use.  Input  e%'aluation  for  a  State  training 
program  provides  valuable  insight  regarding  the  compe- 
tency level  of  the  State  cadre  of  trainers.  If  they  do  not 
have  the  skills  necessary-  to  present  a  training  program, 
the  quality  of  training  is  compromised  at  the  outset. 

When  to  Use  Input  Evaluation 
Valuable  information  can  be  provided  through  input 
evaluation,  enabling  the  program  planner  to  design  pro- 
grams and  pro\-ide  resources  necessary  to  ensure  the  suc- 
cess of  the  program.  Even  the  brightest  students  cannot 
succeed  if  the  program  design  is  faulty  or  if  the  appro- 
priate type  of  equipment  is  not  provided.  Input  evalua- 
tion can  help  the  planner  avoid  these  types  of  costly 
mistakes  or  can  provide  data  for  modifications  to  avoid 
them  in  the  future. 
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As  part  of  a  needs  assessment,  input  evaluation  can  be  used  to  help  the  NET  Coordinator  deter- 
mine whether  resources  are  adequate  for  the  job  to  be  done,  and  whether  the  most  appropriate 
resources  are  selected  and  made  available  for  use  in  the  program.  Further,  it  helps  ensure  that  an 
appropriate  curriculum  or  instructional  design  is  used,  that  instructors  are  selected  on  the  basis  of 
their  qualifications  to  present  the  specific  program,  and  that  students  have  the  appropriate  pre- 
requisite abilities  to  enter  and  succeed  in  the  program  as  it  is  designed. 


When  input  evaluation  is  conducted  during  implementation  (formative)  of  a  project  or  program, 
it  can  provide  feedback  on  changes  that  need  to  be  made  immediately  to  help  remedy  deficiencies 
and  ensure  success.  When  input  evaluation  is  used  at  the  end  of  a  program  (summative),  it  can 
provide  some  revealing  clues  as  to  why  some  programs  failed  while  others  succeeded.  Some  exam- 
ples of  input  evaluation  questions  for  a  school  nutrition  education  program  are  shown  below. 

"Did  the  curriculum  guide  include  the  types  of  instructional  activities 

and  design  to  ensure  student  success  in  meeting  the  objectives 

of  the  nutrition  education  program  for  grades  4-6?" 

"What  kinds  of  instructional  materials  were  most  appropriate  for  use  in  this  program?" 
"Did  the  type  of  equipment  and  facilities  provided  help  ensure  success  in  meeting  the 
objectives  of  the  program?" 

"Did  the  teachers  have  the  appropriate  skills  to  use  the  curriculum  guide  and  the  equip- 
ment and  facilities  provided?" 

"Did  the  students  have  the  requisite  skills  when  they  entered  the  program  to  ensure  they 
were  able  to  succeed?" 


How  to  Design  Input  Evaluation 

Input  evaluation  data  may  be  obtained  during  implementation  (formative)  or  at  the  end  of  a  pro- 
gram or  initiative  (summative).  It  can  be  used  with  a  qualitative  design  using  methods  such  as 
observation,  interview,  ethnography,  expert  judgment,  or  a  quantitative  experimental  or  quasi- 
experimental  evaluation  design.  Giaidelines  for  using  observations  and  interviews  are  described  in 
the  Appendix  of  the  NET  Needs  Assessment  Guide  (1994,  pages  71-87). 

Expert  judgment  may  be  used  as  an  evaluation  method  appropriate  for  input  evaluation.  One 
approach  is  an  expert  review  from  an  individual  who  is  given  an  open-ended  opportunity  to  assess 
the  probability  of  a  program's  success  given  the  input  factors  assigned  to  it.  Another  approach 
would  involve  several  individuals  working  independently  to  make  an  assessment.  Expert  judg- 
ment may  also  be  obtained  from  a  group  of  individuals  using  a  structured  technique  such  as  the 
Delphi  technique  (Cotler,  1988,  p.  273). 

An  experimental  design  could  be  used  appropriately  for  input  evaluation,  however,  this  type  of 
evaluation  entails  a  more  sophisticated  approach  than  most  other  methods  described  in  this 
Guide.  This  type  of  evaluation  design  is  usually  used  by  researchers  in  a  university  setting  who 
have  been  contracted  to  develop  and  manage  the  evaluation.  For  example,  if  the  evaluation  focus 
was  on  the  design  of  a  course,  a  comparison  might  be  made  between  the  use  of  multi-media  in 
one  class  and  a  more  traditional  course  design  in  another.  Using  an  experimental  design  incorpo- 
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rating  a  control  group  and  an  experimental  group,  comparison  could  be  made  of  the  differences 
realized  beru'een  the  groups  based  on  one  group's  use  of  multi-media  materials  (e.g.  audio,  video, 
animation  graphics)  in  a  training  program  as  compared  to  the  other  group's  use  of  traditional 
printed  materials  (a  training  workbook).  An  experimental  design  may  focus  on  a  training  pro- 
gram that  incorporates  experiential  learning  activities  in  every  class  session  as  compared  to  one 
that  incorporates  only  reading,  lectures,  and  class  discussions.  For  an  explanation  of  experimental 
design,  refer  to  a  textbook  on  research  methods  such  as  Understanding  Research  Methods  (1983)  or 
Experimental  Design  (1982). 

Summar\- 

Input  evaluation  provides  information  on  how  to  best  use  resources  to  accomplish  program  goals 
and  objectives.  These  data  can  be  obtained  during  implementation  of  a  program  or  project  (for- 
mative) or  at  the  end  (summative).  Qualitative  and  quantitative  data  collection  methods  can  be 
used  appropriately  for  input  evaluation. 


a. 


Nutrition  Educauon  &  Training  Program  U.S.  Etepartment  of  Agriailture 


Chapter  VII 
Explanation  of  Process  Evaluation 


When  to  Use  It 

During  implementation  or  after 
program  completion: 
^  to  identify  problems  in  procedural 
design, 

to  identify  how  well  the  students 
are  performing  or  performed  on 
interim  performance  objectives, 
and 

Sg  to  assess  to  what  extent  the 
instructor  or  trainer  is  using  or 
used  the  materials  as  they  were 
designed. 

How  to  Design  It 
Qualitative  methods  include: 
^  Direct  observation,  and 
^  Expert  judgment. 
Quantitative  methods  include: 
^  Survey  questionnaires, 
^  Structured  interviews, 
^  Tests, 

^  Authentic  performance  assess- 
ment, and 
^  Experimental  design. 

Who  is  Targeted? 

^  Program  administrators  and 
management  personnel; 

^  Individuals  involved  in 

implementation,  e.g.  trainers  in  a 
staff  development  project;  and 

^  Program  participants. 


Introduction 

Process  evaluation  focuses  on  whether  the  resources 
for  the  program  are  used  appropriately  during  the 
implementation  phase,  if  the  program  is  implement- 
ed as  it  was  designed,  and  if  the  students  are  meeting 
interim  performance  objectives.  Whereas  input  evalu- 
ation addresses  the  appropriateness  and  adequacy  of 
resources,  process  evaluation  asks  whether  they  are 
used  as  they  were  designed.  If  the  focus  is  on  the 
evaluation  of  a  training  program  for  staff  or  a  nutri- 
tion class  for  students,  process  evaluation  is  most 
closely  aligned  with  the  instructional  dimension  of 
the  program.  If  the  focus  is  on  resource  utilization, 
however,  process  evaluation  may  be  designed  to  focus 
on  the  management  of  the  program  as  well  as  on  the 
instructional  process. 

Why  Address  this  Type  of  Evaluation? 

Process  evaluation  gets  at  the  "heart  and  fiber"  of  the 
program  and  helps  provide  answers  as  to  what  specific 
features  during  program  implementation  contributed 
to  success  or,  conversely,  why  the  plans  went  awry. 
This  type  of  evaluation  describes  what  actually  takes 
place  during  the  program  or  project.  Therefore,  it  can 
provide  valuable  information  for  making  mid-course 
corrections  in  a  program  or  project  in  a  timely  manner. 

For  example,  a  one-day  training  program  for  Child  and 
Adult  Care  Food  Program  (CACFP)  Sponsors  offered  in 
eight  locations  throughout  the  State  over  a  year's  time 
should  be  designed  so  that  process  evaluation  data  are 
obtained  after  each  one-day  program  offering.  These  data 
can  help  planners  make  necessary  changes  in  the  pro- 
gram design  during  implementation  in  order  to  improve 


its  effeaiveness.  Thus,  if  after  the  program  is  presented 
twice,  the  process  evaluation  indicates  that  the  trainers  are  not  using  the  training  plan  as  it  was 
designed,  the  NET  Coordinator  should  meet  with  the  trainers  to  ensure  that  changes  are  made.  This 
may  involve  some  re-training  or  coaching  of  the  trainers. 
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One  of  the  major  benefits  or  contributions  of  process  evaluation  is  the  identification  of  flaws  or 
problems  in  procedural  design,  whether  the  design  relates  to  the  instructional  process  or  to  the 
managerial  aspect  of  the  program.  It  helps  provide  an  answer  to  the  question,  "If  we  had  it  do  to 
over  again,  what  would  we  do  differently?"  The  answer  to  this  question  can  provide  valuable 
input  in  the  redesign  of  a  program  or  management  procedure. 

Process  evaluation  lends  itself  to  both  quantitative  and  qualitative  evaluation  methods. 
Traditionally,  most  of  the  process  evaluation  studies  have  utilized  quantitative,  experimental 
designs  where  the  primar\-  objective  was  the  measurement  of  differences.  Consider  a  nutrition 
education  project  which  focuses  on  students  acquisition  of  new  knowledge  about  nutrition  and 
how  nutrition  affects  health.  The  evaluation  design  includes  pre-  and  post-test  measures  for  a 
given  unit  of  study.  The  process  evaluation  design  for  this  project  might  include  explicit  mea- 
sures on  how  frequently  specific  instructional  materials  were  used,  in  what  manner  these  materi- 
als were  used,  with  what  age  groups  of  students  certain  materials  were  used,  what  vypes  of 
instructional  competencies  teachers  needed  to  use  the  materials  most  effectively,  and  what  differ- 
ences these  factors  had  on  the  student's  acquisition  of  new  knowledge. 

Qualitative  designs,  on  the  other  hand,  might  focus  more  on  the  feelings,  attitudes,  general  con- 
cerns, and  issues  that  surround  and  or  permeate  the  process,  and  could  provide  meaningful  clues 
as  to  why  the  process  is  or  is  not  working.  For  example,  the  focus  might  be  on  the  importance  of 
multi-cul rural  sensitivity  and  its  eft'ect  on  the  attitudes  of  students  in  nutrition  education  classes. 
As  our  population  becomes  more  diversified  in  its  ethnicity,  it  is  important  that  ethnic  differ- 
ences, cultures,  and  customs  be  recognized  and  valued.  Thus,  when  the  Dietan*  Guidelines  are 
taught,  an  evaluation  question  might  ask  how  teachers  explain  the  Food  Guide  Pyramid  using 
food  examples  which  illustrate  the  variety  of  ethnic  backgrounds  of  their  students.  This  question 
illustrates  the  process  used  in  the  classroom  to  present  currioolum  materials. 

When  to  Use  Process  Evaluation 

Process  evaluation  always  addresses  what  happens  during  the  implementation  phase  of  a  program 
or  project.  The  data  may  be  gathered  during  the  implementation  phase  or  they  may  be  obtained 
after  the  implementation  phase.  In  either  case,  the  evaluation  question  asks  "how  or  what"  about 
the  process  of  implementation.  Following  are  some  examples  of  process  evaluation  questions 
w^hich  could  be  addressed  during  implementation  or  following  implementation. 

^  "How  are  the  aarriculum  materials  being  used  in  the  instnictional  process?" 

(formative  evaluation) 
^  "How  were  the  airriculum  materials  used  in  the  instructional  process?" 
(summative  evaluation) 

^  "How  are  the  equipment  and  facilities  being  used  in  the  process  of  implementing  the 
program?"  (formative  evaluation) 

^  "How  were  the  equipment  and  facilities  used  in  the  process  of  implementing  the  pro- 
gram?" (summative  evaluation) 
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2£  "Is  the  instructor  utilizing  the  varien.'  of  instructional  techniques  appropriate  for  pre- 
senting the  program?"  (formative  evaluation) 

2£  "Did  the  instruaor  utilize  the  variet\-  of  instruaional  techniques  appropriate  for  pre- 
senting the  program?"  (summative  evaluation) 

"How  well  are  the  students  performing?  Are  they  meeting  interim  performance 
objectives?"  (formative  evaluation) 
3£  "How  well  did  the  students  penorm  on  the  interim  performance  objectives?'"  (sum- 
mative evaluation) 

The  focus  of  process  evaluation  might  also  center  on  other  aspects  of  the  overall  NET  Program.  It 
might,  for  example,  focus  on  the  management  of  the  program  itself.  From  this  perspective, 
process  evaluation  for  the  NET  Program  might  address  such  questions  as: 
2£  "How  well  are  we  staying  within  budget?" 

2£  How  well  are  we  following  appropriate  procedures  in  managing  external  contracts  to 
ensure  delivery  of  contracted  products  and  services?" 

How  to  Design  Process  Evaluation 

Since  process  evaluation  assesses  what  takes  place  during  implementation  of  a  project  or  program, 
a  varier.-  of  forms  of  feedback  can  be  used.  The  methods  used  most  often  include  written  ques- 
tionnaires, tests,  interviews,  and  observations.  Assessment  of  interim  student  performance, 
whether  for  a  K-12  classroom  or  for  an  adult  learning  situation,  is  an  important  type  of  process 
evaluation. 

Traditionally,  we  have: 

2£  Compared  learners  to  each  other  and  placed  more  importance  on  those  comparisons 

than  on  what  was  actually  learned, 
2£  Used  paper  and  pencil  tests  and  checklists  to  evaluate  what  students  learned,  and 
2£  Excluded  students  from  the  evaluation  process. 
Contemporary  assessment  of  performance  requires  tis  to: 

Have  the  students  take  an  active  role  in  evaluating  their  own  progress, 
2£  Use  a  variety  of  sources  and  tools  to  determine  what  students  have  learned,  and 
3£  Evaluate  students  in  relation  to  authentic  criteria  of  actual  performance. 

There  are  several  types  of  process  evaluation  lased  in  student  classrooms  and  adult  training  sinia- 
tions.  For  K-12  students,  the  move  today  is  away  from  paper- and-pencil  tests  for  either  process  or 
product  evaliaation,  and  toward  the  use  of  evaluations  where  the  learner  can  demonstrate  use  of 
the  new  knowledge  and  skills.  This  move  to  a  more  authentic  type  of  assessment  is  also  true  for 
adult  training  programs.  Performance  assessment  is  one  of  the  most  useful  types  of  evaluation  in 
student  classrooms  as  well  as  for  job-related  training.  This  type  of  assessment  requires  the  learner 
to  acnoally  demonstrate  the  skill  or  use  of  the  knowledge.  For  K-12  students  as  well  as  for  adult 
learners,  this  type  of  assessment  is  more  appropriate  than  paper-and-pencil  tests  because  it 
requires  the  learner  to  demonstrate  learning  in  a  situation  similar  to  the  one  in  which  it  will 
actually  be  used. 
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Authentic  performance  assessments  may  take  many  forms,  such  as: 

•  models, 

•  videos, 

•  debates, 

•  exhibits, 

•  diagrams, 

•  research  projects, 

•  teacher  observation, 

•  anecdotal  records, 

•  community  projects, 

•  structured  activities/projects, 

•  demonstrations, 

•  displays, 

•  reports, 

•  journals, 

•  simulations, 

•  projects, 

•  presentations, 

•  case  problems,  and 

•  concrete  representations. 


The  move  today,  for  student  classroom  instruction  and  adult  learning  situations,  is  to  authentic 
performance  assessment.  This  type  of  assessment  focuses  on  the  identification  of  evaluative  crite- 
ria and  strategies  to  determine  whether  or  not  a  student  can  actually  demonstrate  his/her  profi- 
ciency to  perform  a  given  objective  in  a  real-life  context.  Standardized  tests  or  the  typical  true- 
false,  matching,  completion,  or  multiple-choice  tests  usually  assess  student  learning  at  the  com- 
prehension level  of  cognition,  often  requiring  only  recall  of  facts.  Authentic  assessment,  however, 
focuses  on  the  use  of  criterion-referenced  tests  or  evaluation  measures  which  are  designed  to  assess 
learning  at  a  higher  level,  from  the  application  up  to  the  evaluation  level.  Criterion-referenced 
measures  require  absolute  congruence  with  the  performance  objective  and  objectivity  so  as  to 
ensure  a  common  understanding  between  the  student  and  the  instructor  as  to  what  is  being  actu- 
ally required  and  the  measures  by  which  the  performance  will  be  assessed. 

Use  of  alternative  assessments  in  schools  represents  a  fundamental  shift  from  reliance  on  stan- 
dardized testing  techniques  that  have  characterized  student  assessment  for  decades  (Wolf,  Bixby, 
Glen,  &  Gardner,  1991).  Although  there  are  various  alternative  assessment  techniques,  they  are 
generally  characterized  by: 

^  The  student  developing  products  rather  than  recognizing  answers, 

^  The  student  demonstrating  a  depth  of  knowledge  rather  than  breadth,  and 

^  The  teacher  using  his/her  expertise  to  judge  the  assessment  rather  than  using  a  specific 

mechanical  standard  (Calfee  and  Perfumo,  1993). 
These  characteristics  could  be  used  as  guidelines  for  developing  authentic  assessment  measures  for 
both  K-12  students  and  adult  learners. 
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One  of  the  most  popular  forms  of  alternative  assessment  is  the  use  of  portfolios.  Portfolio  assess- 
ment is  appropriate  to  use  for  process  evaluation  and  for  product  evaluation.  Refer  to  Appendix  D 
for  a  description  of  portfolio  assessment. 

Summar\- 

Process  evaluation  provides  feedback  about  program  implementation.  It  is  usually  conducted  dur- 
ing implementation,  but  may  also  examine  the  process  of  implementation  afiter  the  program  or 
project  has  been  completed.  Assessment  of  interim  student  performance  is  an  important  aspect  of 
process  evaluation.  One  of  the  most  effective  approaches  to  use  for  this  t\-pe  of  evaluation  is 
authentic  performance  assessment. 
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Chapter  VIII 
Expla72atio7i  of  Product  Evaluation 


When  to  Use  It 
Following  program  implementation 
to  describe  students  performance  in 
relation  to  stated  performance  objec- 
tives (measurable  outcomes). 

How  to  Design  It 

C  Authentic  performance  assess- 
ment, 
P Tests, 

P  Rating  scales, 

P  Attitude  questionnaires, 

P!Obser\"ation. 

PCase  study,  and 

C  Experimental  design. 

Who  is  Targeted? 

C  Adult  training  program  partici- 
pants, and 
P!K-12  students. 


Introduction 

Product  e\'aluation  focuses  on  student  outcomes,  abilit\- 
le\-els,  and  performance  after  the  completion  of  a  given 
program.  This  t}-pe  of  evaluation  may  be  referred  to  as 
outcome  e%-aluation.  Instruaional  designers  sometimes 
define  performance  objectives  as  interim  or  enabling  objec- 
tives, which  would  be  evaluated  in  process  evaluation,  and 
terminal  objectives,  which  would  be  evaluated  in  produa 
e\"aluation.  Regardless  the  name  it  is  given,  this  type  of 
e\'aluation  addresses  what  students  know  and  are  able  to 
do  upon  completion  of  the  program.  Perhaps,  more 
importantly,  it  focuses  on  the  degree  to  which  students  are 
able  to  apply  the  new  knowledge  in  real  life  settings. 
Likewise,  produCT  evaluation  fctcuses  on  what  staff  person- 
nel know  and  are  able  to  do  upon  the  completion  of  a  staff 
development  training  program.  Thus,  product  e^-aluarion 
seeks  to  determine  the  demonstration  of  behavioral 
change. 


Why  Address  this  T\-pe  of  Evaluation? 
Unless  product  evaluation  is  viewed  as  one  of  the  most 
essential  components  within  a  multi-faceted  evaluation 
design,  no  real  rele\'ancy'  or  meaning  can  be  given  to  context,  input,  and  process  evaluation.  When  con- 
text, input,  and  process  evaluation  are  determined  at  the  end  of  a  program  or  project  (summatively),  the 
data  are  examined  in  relation  to  student  outcomes  and/or  the  outcomes  of  the  program.  Some  examples 
are  shown  below. 

A  context  evaluation  question:  'T)id  this  type  of  administrative  support  or  environment 
contribute  to  the  success  of  the  program?" 

An  input  evaluation  question:  "Did  these  types  of  resources  and  the  instructional 
design  contribute  to  the  success  of  the  program?" 

A  process  evaluation  question:  "Did  the  implementation  of  the  program  in  rhi";  manner 

contribute  to  the  success  of  the  program?" 
Thus,  product  evaluation  provides  critical  information  for  jtidging  the  success  of  a  program  or  project. 
Knowing  that  the  administrative  staff  were  supportive,  that  the  program  was  needed,  that  all  of  the  req- 
uisite inputs  were  provided,  and  that  the  program  was  implemented  as  planned  still  does  not  address 
the  question:  "But  can  the  students,  staff  actually  do  these  things?"  That  question  can  only  be  ansu-ered 
through  a  carefully  designed  product  evaluation. 
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When  to  Use  Product  Evaluation 

Just  as  evaluation  doses  the  loop  in  the  Assessment-Planning-Budgeting-Implementation- 
Evaluation  Process,  product  evaluation  closes  the  loop  in  the  curriculum  and  instructional  design 
process.  The  curriculum  planner  or  instructional  designer  begins  with  an  assessment  of  what  stu- 
dents need  to  know  and  are  demonstrating  as  their  vrpical  behavior  or  performance  in  comparison 
to  what  they  should  know  and  demonstrate  in  their  daily  lives  or  job  performance.  Based  on  that 
assessment,  the  design  phase  of  the  instructional  process  is  begun.  Once  designed,  the  curriculum 
is  implemented  through  the  instructional  process,  and  the  ultimate  outcomes  of  that  process  are 
assessed.  The  ultimate  performance  outcomes  are  assessed  as  product  evaluation.  Based  on  the 
evaluation  of  the  students"  performance  upon  completion  of  the  program,  the  needs  assessment 
begins  again  and  the  curriculum  is  re-designed  or  modified  for  improvement. 

How  to  Design  Product  Evaluation 

Product  evaluation  focuses  on  what  the  students  can  actually  do.  This  raises  two  critical  questions: 
"What  level  of  learning  or  cognition  is  required?"  and  "When,  or  at  what  point  in  time  after  pro- 
gram completion,  will  student  performance  or  behavioral  change  be  evaluated?"  The  answer  to 
the  first  question  requires  a  familiarir,'  with  the  levels  of  learning  in  the  cognitive  domain,  as 
well  as  familiarit}'  with  alternative  application  designs.  The  study  of  cognition  is  not  an  exact  sci- 
ence because,  as  yet,  we  do  not  truly  understand  how  the  brain  and  the  mind  work.  However, 
most  learning  theorists  and  curriculum  planners  recognize  the  existence  of  a  learning  continuum 
or  taxonomy,  within  which  several  levels  of  learning  have  been  generally  described. 
To  illustrate  the  learning  continuum  or  taxonomy,  consider  an  example  of  a  student  learning 
experience  that  is  focused  on  the  Food  Guide  Pyramid  (See  Illustration  5  •  An  example  of  the 
learning  continuum  with  a  focus  on  the  Food  Guide  Pyramid.). 
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Evaluation. 
/The  student 
may  be  asked 
to  make  several 
recommendations 
for  impromg  the 
weekly  menu  so  as  to 
/assure  the  inclusion  of  the 
recommended  number  of 
senlngs  even,'  day  and  to 
assure  variet}"  in  each  food  group 
e\en,'  day. 

/Synthesis^  The  student  may  be  given  a 
/weekly  menu  from  the  cafeteria  to  deter- 
/mine  the  composite  picture  for  the  week, 
/including  the  identification  of  the  variety"  of 
foods  used  to  meet  the  recommended  number  of ^ 
senlngs  on  the  various  levels  of  the  Food  Guide 
PATamid. 

/■\nalvsis.  When  gi\'en  a  sample  daily  menu,  the  student 
/would  be  able  to  determine  whether  the  daily  menu  is 
/consistent  ^ilh  the  recommended  number  of  senlngs  on 
/the  various  levels  of  the  Food  Guide  PATamid. 
^/Application.  The  student  is  able  to  use  the  information  or  knowl- 
/edge  in  some  limited  way.  e.g.  identif\"  what  he  she  might  eat  for 
one  dav  to  include  at  least  one  selection  from  each  level  of  the  Food 


/n 


/ 

/Guide  F^Tamid. 

/Comprehension.  The  student  may  recall  definitions  and  recognize  exam-  \ 
'^ples  of  or  differences  betu'een  categories  or  groups,  e.g.  match  examples  of  ■ 
'different  foods  from  a  given  list  to  the  appropriate  food  groups  in  the  Food 
'Guide  PATamid. 

'Knowledge  is  the  lowest  level  of  cognition.  The  student  is  only  able  to  recall  facts 
'e.g.  name  the  groups  of  foods  in  the  Food  Guide  PATamid. 
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This  example  shows  that  the  knowledge  level  of  the  student  and  the  ability  to  apply  the  knowl- 
edge increases  at  each  level  or  phase  along  the  continuum.  Thus,  if  the  student's  performance 
level  is  expected  to  change,  the  curriculum  and  instructional  design  must  move  the  student  at 
least  to  the  application  level  or  higher.  Curriculum  designs  and  tests  which  focus  solely  on  the 
recall  of  facts  and  on  such  outcomes  as  naming,  listing,  describing,  defining,  and  matching,  will 
never  effect  behavioral  change.  Meaningful  product  evaluation,  therefore,  must  focus  on  what  the 
students  can  do  at  the  application  level  of  cognition  or  higher. 

Another  aspect  of  product  evaluation  is  that  of  application  designs,  which  require  the  curriculum 
planner  to  examine  the  variety  of  contexts  within  which  the  application  of  learning  should  be 
demonstrated.  Again,  there  are  several  levels  of  contexts,  each  requiring  a  higher  level  of  profi- 
ciency. The  example  below  shows  several  different  contexts,  each  requiring  a  higher  level  of  profi- 
ciency to  be  demonstrated  by  a  school  nutrition  manager  in  a  training  course  on  implementing 
the  Dietary  Guidelines. 


Application  is  demonstrated  in  the  abstract  with  little  or  no  relationship  to  real 
problems  or  situations,  e.g.  simple  addition  of  the  fat  grams  in  the  foods  in  a 
meal  given  as  a  written  math  problem. 

Application  is  demonstrated  within  a  meaningful  context,  e.g.  calculation  of  the 
total  fat  content  for  a  real  meal  that  an  individual  student  selected  and  ate  on  a 
specific  day. 

Application  is  demonstrated  within  a  meaningful  problem-solving  context,  e.g. 
after  the  manager  determines  whether  the  example  student  ate  a  large  amount  of 
fat  in  the  selected  lunch,  the  manager  is  asked  to  outline  changes  he/she  can  make 
in  the  menu  offerings  and  ways  the  food  is  prepared  to  resolve  that  problem. 
Application  is  demonstrated  within  a  different,  but  relevant,  problem-solving 
context,  e.g.  the  manager  may  find  that  he/she  has  a  high  cholesterol  problem  (or 
other  type  of  relevant  health  problem)  and  be  asked  to  outline  substantive 
changes  to  be  made  in  his/her  daily  eating  habits  to  reduce  fat  and  promote 
improved  nutritional  health. 


If  the  instructional  designer  wants  to  effect  substantive  behavioral  change  on  the  part  of  the 
learners,  the  design  of  the  curriculum  must  guide  the  students  to  the  level  of  performance  actual- 
ly required  in  a  real  life  setting.  An  analogous  case  or  argument  can  be  made  in  staff  develop- 
ment, where  the  focus  is  on  training  instructors  or  food  service  personnel.  If  the  objective  is  to 
change  the  demonstrated  performance  or  behavior  of  the  staff,  the  training  program  must  be 
designed  accordingly.  Since  product  evaluation  must  be  tightly  interfaced  with  instructional 
design,  and  the  terminal  objectives  included  therein,  the  product  evaluation  must  focus  on  what 
the  student  or  staff  member  can  do  in  the  context  in  which  that  behavior  or  performance  is 
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identified  in  the  curriculum  objectives.  Thus,  for  the  child  nutrition  professional,  the  context  for 
performance  assessment  is  the  actual  job  situation.  This  is  the  true  essence  of  authentic  perfor- 
mance assessment. 

The  second  question  the  instructional  designer  must  also  consider  is  the  time  factor  in  regard  to 
when  product  evaluation  should  occur  ("When,  or  at  what  point  in  time  after  program  comple- 
tion, will  student  performance  or  behavioral  change  be  evaluated?").  St. Pierre  (1982)  refers  to  this 
as  "proximity  of  outcomes."  Should  the  evaluation  be  conducted  immediately  upon  completion  of 
the  instruction  or  program,  or  should  the  evaluation  be  conducted  at  some  distant  time  in  the 
future?  In  relation  to  product  evaluation,  the  answer  is  immediately  upon  the  completion  of  the 
instruction  or  program.  This,  then,  leaves  the  question  of  immediacy  to  be  defined. 
Unfortunately,  there  is  no  clearly  delineated,  objective  definition  which  can  be  used  in  every  situ- 
ation. The  product  evaluation  may  be  on  the  last  day  of  class,  the  day  after  the  course  or  program 
has  been  completed,  or  six  months  after  program  completion.  Perhaps  the  definition  can  most 
appropriately  be  determined  by  answering  the  question  "can  they?"  "Can  the  learners  demon- 
strate the  level  of  proficiency  required?"  And  "how  does  one  know  that  they  can?" 

When  performance  objectives  are  written  in  measurable,  understandable  terms,  the  instructional 
designer  has  essentially  defined  what  performance  must  be  demonstrated  by  the  learners.  The 
main  decision  left  is  to  determine  the  most  effective  way  to  measure  the  level  of  performance  of 
the  learner.  Product  evaluation  determines  the  level  of  student  proficiency  on  defined  perfor- 
mance objectives  and  can  be  designed  using  the  same  methods  of  evaluation  as  used  for  process 
evaluation.  Authentic  performance  assessment,  one  of  the  most  useful  types  of  evaluation  has 
already  been  described  in  Chapter  7,  Explanation  of  Process  Evaluation.  Portfolio  assessment,  one 
of  the  most  popular  forms  of  authentic  assessment,  is  described  in  Appendix  D. 

Summary 

Product  evaluation  measures  the  attainment  of  performance  objectives  (measurable  outcomes). 
When  objectives  are  written  in  measurable,  understandable  terms,  the  product  evaluation  has 
essentially  been  defined.  One  of  the  most  useful  methods  for  product  evaluation  in  the  classroom 
or  adult  staff  development  program  is  authentic  performance  assessment. 
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Chapter  IX 
Explanation  of  Impact  Evaluation 


When  to  Use  It 

As  an  examination  of  the  long-term 
impact  of  the  program. 

How  to  Design  It 

To  evaluate  the  impact  of  training  pro- 
grams on  behavior  of  participants: 

"4"  Interviews, 

"I" Focus  groups, 

~4~Self-assessment  questionnaires, 

~4~Obser\'ations, 

~|~Case  studies,  and 

"I'Portfolio  assessment. 
To  evaluate  the  impact  of  training  pro- 
grams on  the  organization: 

"^"Interviews, 

"ii" Focus  groups, 

~;^"  Analysis  of  documents, 

"^"Observations, 

"i^"Case  studies,  and 

"1" Cost-benefit  studies. 
To  evaluate  the  impact  of  nutrition  edu- 
cation programs  on  students'  food-relat- 
ed behavior  at  school  and/or  knowledge 
and  attitudes  about  food: 

"|"Observation, 

"i^" Attitude  survey, 

"i"Criterion-referenced  test  of  food 
and  nutrition  knowledge,  and 

"1"  Analysis  of  records  or  documents. 

Who  is  Targeted? 

~i~ Program  administrators  and  man- 
agement staff, 
~i~  Adult  program  participants, 
"ii" Participants'  supervisors, 
"("Students, 
Parents, 

~]i"School  faculty  and  administrators, 
";i" Members  of  the  community,  and 
"4"  Individuals  from  cooperating  agencies. 


Introduction 

Impact  evaluation  focuses  on  the  cumulative  differ- 
ence the  NET  Program  makes  on  the  target  groups 
and  on  the  issues  or  problems  it  is  intended  to 
address.  Its  focus,  therefore,  is  on  long-term  impact. 
Although  this  type  of  evaluation  is  the  most  complex 
in  design  and,  to  some,  the  most  difFicult  to  conduct, 
it  often  provides  the  basis  for  critical  p>olicy  decisions 
which  can  affect  the  continued  life  and  substance  of  a 
program. 

Why  Address  this  T)^e  Of  Evaluation? 
This  type  of  evaluation,  more  than  any  other,  can 
provide  the  justification,  or  lack  thereof,  for  critical 
policy  decisions,  including  the  continuation  of  a  pro- 
gram. "Proof  of  value,"  is  the  essence  of  impact  eval- 
uation. 

Some  States  and  some  national  administrations  have, 
over  the  years,  used  a  "zero-based  budgeting"  strate- 
gy as  a  basis  for  their  decision-making  process.  This 
strategy  requires  program  managers  or  directors  to 
justify  why  their  programs  should  be  continued  prior 
to  submitting  a  budget  request.  The  primary  and 
most  legitimate  focus  of  that  justification  must  be 
program  impact  or  proof  of  value.  When  administra- 
tions change,  as  they  inevitably  and  continually  do, 
most  will  begin  with  a  search  for  ways  to  cut  the  cur- 
rent budget  in  order  to  find  the  resources  needed  to 
support  new  programs  which  they  have  pledged  to  ini- 
tiate. Again,  evidence  of  value,  evidence  of  the  differ- 
ence the  program  has  made  and  is  making,  and  proof 
of  the  program's  impact  on  the  alleviation  of  specific 
societal  problems  assumes  critical  importance.  These 
data  can  only  be  provided  through  impact  evaluation. 
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When  to  Use  Impact  Evaluation 

The  scope  of  impact  evaluation  can  be  quite  extensive,  ranging  from  a  behavior  change  in  adult 
participants  in  staff  development  programs,  to  changes  in  the  Child  Nutrition  Programs,  to 
changes  in  the  students'  food-related  behavior  at  school.  Since  real  change  takes  time,  a  NET 
Coordinator  will  not  be  concerned  with  determining  the  impact  of  a  two-week  unit  of  study 
incorporated  into  the  eighth  grade  curriculum  of  a  school,  nor  the  impact  of  a  one-day  or  even  a 
several-day  staff  development  session.  Rather,  impact  evaluation  takes  a  more  holistic  look  at  the 
overall  effect  of  an  entire  program  of  activity,  e.g.,  the  cumulative  impact  of  all  NET  Program 
activities  within  a  particular  school  district,  or  region,  or  State  over  the  course  of  an  extended 
period  of  time.  Impact  evaluation,  thus,  speaks  to  the  other  side  of  the  "proximit}'"  issue  which 
was  raised  previously  in  product  evaluation.  The  time-line  for  the  evaluation  is  extended  from 
"immediately"  after  program  completion  to  a  more  "distant  future." 

How  to  Design  Impact  Evaluation 

Impact  evaluation,  like  context  evaluation,  lends  itself  to  qualitative  evaluation  design  as  well  as 
to  quantitative  design.  Because  of  the  wide  array  of  variables  affecting  both  the  type  and  exten- 
siveness  of  the  identified  impact,  an  empirical  approach  which  seeks  to  prove  direct  cause  and 
effect  scenarios  may  be  difficult,  if  not  impossible.  Thus,  attention  should  also  be  given  to  deter- 
mining "indications  of  impact"  for  which  a  qualitative  evaluation  design  can  be  quite  helpful. 

For  example,  consider  an  evaluation  study  designed  to  determine  the  impact  of  a  variety  of  class- 
room and  cafeteria  NET  activities  on  students'  knowledge,  attitudes,  and  food  practices  in  regard 
to  using  the  principles  of  the  Dietar\-  Guidelines.  Proving  direct  cause  and  effect  would  be  virtu- 
ally impossible,  as  well  as  inappropriate,  because  of  the  wide  array  of  uncontrolled  variables  that 
affect  students'  knowledge,  attitudes,  and  food  practices.  Uncontrolled  variables  could  include 
advertising,  parental  influence,  access  to  food,  time  to  eat,  and  many  others.  However,  indications 
of  impact  could  be  obtained  from  the  results  of  student  focus  groups  before  the  nutrition  educa- 
tion classroom  and  cafeteria  activities  on  the  Dietar\'  Guidelines,  and  then  some  months  after  the 
activities  have  been  completed.  In  this  example,  the  impact  evaluation  would  not  isolate  specific 
classroom  or  cafeteria  activities  to  determine  what  difference  they  had  made.  Rather,  the  impact 
evaluation  would  examine  the  food-related  behavior  of  students  at  school  in  a  more  holistic  way. 

Because  of  the  comprehensive  scope  and  extended  time-frame  characteristic  of  impact  evaluation, 
the  evaluation  plan  should  identify^  an  array  of  evaluation  questions  which  could  be  examined 
over  a  period  of  time.  Changes  in  the  knowledge,  skills,  and  attitudes  of  adult  participants  of 
staff  development  programs  can  be  measured  by  evaluation  methods  such  as  interviews,  focus 
groups,  self-assessment  questionnaires,  observations,  case  studies,  and  portfolio  assessments. 
Changes  in  the  organization,  whether  a  school  nutrition  program,  a  center,  or  a  feeding  site,  can 
be  measured  through  interviews,  questionnaires,  analysis  of  documents,  observations,  case  studies, 
and  cost-benefit  studies.  Quantitative  changes  in  students'  food-related  behavior  at  school  can  be 
measured  through  obser\'ation,  a  criterion-referenced  test  of  food  and  nutrition  knowledge,  and 
an  analysis  of  records  such  as  the  food  production  record. 
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Summar\' 

Impact  evaluation  is  "proof  of  value"  evaluation.  It  addresses  the  cumulative  differences  the  pro- 
gram has  made.  The  focus  is  on  long-term  impact.  Like  context  evaluation,  impaa  evaluation 
questions  can  usually  be  answered  by  collecting  data  using  qualitative  methods. 
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Appendix  A 

Guidelines  for  Developing  Evaluation  Instruments 

The  development  of  instruments  for  evaluation  involves  the  use  of  accepted  design  principles. 
Phillips  (I99I,  pp.  82-87)  identifies  six  important  principles  to  be  addressed  during  the  develop- 
ment of  evaluation  instruments. 

Validity 

Validity  is  one  of  the  most  important  characteristics  of  an  evaluation  instrument.  A  valid  instru- 
ment measures  what  the  evaluator  wants  to  measure.  The  American  Psychological  Association  has 
adopted  four  approaches  in  determining  the  validity  of  an  instrument  (Phillips,  1991,  pp.  82- 
85). 

□  Content  validity  refers  to  how  well  the  instrument  represents  the  content  of  the  program  to 
be  measured.  This  type  of  validity  is  insured  when  the  important  behaviors  and  information 
covered  are  represented  in  a  balanced  way  in  the  instrument. 

□  Construct  validity  refers  to  the  extent  to  which  the  construct  to  be  measured  is  represented  in 
the  instrument.  Constructs  represent  variables  such  as  attitude,  skill,  or  ability.  Construct 
validity  is  a  complex  issue.  One  of  the  most  efficient  ways  to  determine  construct  validity  is 
to  have  a  group  of  experts  review  the  instrument  and  determine  whether  it  is  an  accurate 
measurement  of  the  construct(s). 

□  Concurrent  validity  means  the  extent  to  which  the  results  from  administering  an  instrument 
agree  with  the  results  of  other  instruments  used  at  approximately  the  same  time  to  measure 
the  same  characteristics.  For  example,  an  attitude  survey  for  CACFP  Sponsors  regarding 
nutrition  education  in  Centers  is  administered  to  a  sample  of  Sponsors.  Another  attitude  sur- 
vey is  administered  to  the  same  group  and  both  surveys  obtain  very  similar  results.  The  sur- 
vey instrument  would  have  concurrent  validity. 

□  Predictive  validity  refers  to  the  extent  an  instrument  can  predict  future  results  or  behaviors. 
This  type  of  validity  would  be  important  for  nutrition  education  programs  where  the  evalua- 
tion question  addresses  predicting  food  related  behavior  (at  school)  using  results  of  evaluation 
of  a  nutrition  education  program. 

Phillips  (I99I,  p.  85)  suggests  some  simple  guidelines  for  improving  validity.  These  guidelines 
can  and  should  be  used  when  developing  any  kind  of  evaluation  instrument. 

1.  Include  an  adequate  number  of  appropriate  items  in  the  instrument.  This  is  a  question  of  bal- 
ance since  too  few  items  compromise  the  validity  and  too  many  become  cumbersome  and 
could  prevent  some  respondents  from  completing  the  form. 

2.  Make  every  effort  to  reduce  response  bias.  People  tend  to  say  what  they  think  their  supervi- 
sors want  to  hear.  Thus,  it  is  important  to  design  instruments  in  which  people  can  give  hon- 
est responses.  Insuring  confidentiality  is  important  for  many  types  of  evaluation. 

3.  Administer  the  instrument  objectively.  Staff  members  or  consultants  who  administer  evalua- 
tion instruments  should  be  trained  to  use  the  type  of  evaluation  technique  in  a  professional 
manner. 
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4.   Recognize  the  weak  link  benveen  attitude  and  behavior.  This  guideline  is  particularly  impor- 
tant in  nutrition  education  and  food  service  training.  It  is  important  to  recognize  that  a  short 
training  workshop  of  a  few  hours  or  a  few  days  may  positively  influence  attitudes,  and  even 
improve  content  knowledge,  but  is  less  likely  to  change  behavior.  Even  though  a  person  may 
indicate  a  certain  attitude,  his/her  behavior  may  be  inconsistent  with  the  attitude.  A  nutri- 
tion education  strand  of  a  health  curriculum  taught  for  several  hours  over  a  six  week  period  in 
one  grade  level  may  positively  influence  a  student's  attitude  and  interest  in  nutrition  and  may 
increase  the  student's  knowledge  of  nutrition.  However,  such  a  program  is  unlikely  to  result 
in  significant  changes  in  food  choices  or  in  nutritional  status. 

Reliability 

Reliability,  another  important  characteristic  of  evaluation  instruments,  means  that  an  instrument 
provides  consistent  measurements  of  an  item.  For  example,  if  Summer  Food  Sen-'ice  Sp>onsors 
were  administered  a  survey  questionnaire  regarding  their  attitude  toward  providing  nutrition 
education  materials  to  students  at  feeding  sites  and  the  same  survey  was  administered  several 
days  later,  a  reliable  instrument  would  obtain  the  same  or  very  similar  results. 

There  are  four  generally  accepted  procedures  for  ensuring  reliabilit)-: 

n  test/retest, 

n   alternate-form  method, 

n   split-half  procedure,  and 

n   inter-item  correlations. 

An  explanation  of  these  procedures  can  be  found  in  textbooks  and  journal  articles  listed  in  the 
References  chapter  of  this  Guide. 

Ease  of  administration 

An  evaluation  that  is  difficult  to  administer  should  not  be  used.  Directions  should  be  simple  and 
easy  to  understand  both  for  the  people  administering  the  instrument  and  for  the  respondents.  A 
field  test  with  a  typical  group  of  respondents  can  provide  the  evaluator  with  important  feedback 
regarding  the  ease  of  administration. 

Simplicity'  and  brevity 

The  rule  of  thumb  here  is  to  be  as  brief  as  possible  to  cover  the  topic  and  cover  it  in  the  simplest 
way.  Length  does  not  equal  effectiveness.  Consider  the  level  of  readability'  of  written  instruments 
and  the  background  of  respondents.  The  readability  level  increases  when  sentences  are  long  and 
multisyllabic  words  are  vised.  To  promote  ease  of  readability  when  developing  data  collection 
instruments: 
n   Be  brief, 

n   Keep  it  as  simple  as  possible,  and 

n   Leave  as  much  white  space  as  possible  on  each  page. 

A  field  test  can  provide  valuable  information  regarding  how  well  the  evaluation  designer  has 
addressed  this  important  consideration. 
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Economy 

A  good  instrument  is  one  that  obtains  the  evaluation  information  specified  by  the  objectives  for 
the  least  cost.  Costs  are  affected  by  design  time,  time  for  administration,  time  for  analysis,  costs 
of  analysis,  and  time  needed  for  developing  the  report.  When  data  that  are  already  available  can 
be  used  for  evaluation,  of  course,  the  cost  is  drastically  reduced.  The  Data  Review  Method  is 
described  in  the  Appendix  of  the  NET  Needs  Assessment  Guide  (1994)  and  offers  a  cost  effective 
method  of  evaluation  in  certain  situations. 

As  with  needs  assessment,  the  NET  Coordinator  must  balance  the  cost  of  obtaining  the  evalua- 
tion information  needed  for  program  management  decisions  and  accountabilit}'  with  the  potential 
usefulness  of  the  data  obtained.  The  development  of  evaluation  instruments  can  be  time  consum- 
ing and  thus,  costly.  However,  some  types  of  evaluation  are  not  costly  at  all.  The  overriding  con- 
sideration is  that  an  investment  in  evaluation  is  often  the  most  critical  investment  in  the  pro- 
gram's future. 
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Appendix  B 
Data  Collection  Method:  Case  Study 

Definition 

The  case  study  is  an  intensive,  detailed  analysis  and  description  of  a  single  program,  class,  insti- 
tution, or  phenomenon  in  the  context  of  its  environment. 

Characteristics 

A  case  study  used  as  a  data  collection  method  for  evaluation  is: 

Specific  -  focuses  on  events  in  one  particular  setting  such  as  a  class  in  management  training,  a 
specific  group  of  new  sponsors,  particular  xypts  of  instructors  such  as  those  who  specialize  in 
train-the-trainer  workshops. 

Holistic  -  tries  to  capture  the  totality-  of  the  phenomenon;  looks  at  each  pan  as  well  as  the  whole; 
looks  at  everything  that  makes  up  that  entity. 

Longitudinal  -  usually  tells  a  ston,-  over  time,  although  at  times  it  may  look  at  a  "slice  of  life"  or 
a  given  instance. 

Qualitative  -  uses  a  variety  of  methods  to  collect  information  including  interviews,  observations, 
diaries,  checklists,  and  data  review  or  records  analysis.  The  emphasis  and  foais  is  on  the  quality 
of  the  specific  information,  not  the  quantir,-.  This  method  does  not  lend  itself  to  "statistical  num- 
bers-crunching." 

Advantages 

n   Problems  can  be  formulated  as  the  evaluation  progresses  since  the  evaluator  is  not  bound  to 
a  priori  questions. 

n   Data  categories  can  be  constantly  modified  making  them  more  suitable  to  analysis. 

n   The  distortion  of  variables  that  may  be  difficult  to  quantify-  can  be  reduced  by  the  evaluator. 

n   Personal  and  immediate  feedback  is  provided. 

n  Needs  of  diverse  people  and  situations  can  be  accommodated. 

Limitations 

n   Although  a  case  study  data  collection  method  may  appear  to  be  simple  to  conduct,  it  requires 
considerable  skill  in  such  areas  as  inter\'iewing,  scientifically  sampling  case  study  sites,  and 
designing  coding  systems  which  allow  for  easy  retrieval  of  information. 

n   The  evaluator  may  fall  into  the  trap  of  looking  at  everything  and  thereby  losing  focus.  This 
creates  a  difficult  siniation  in  which  to  summarize  pertinent  information  and  findings. 
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□   Case  study  reports  tend  to  be  lengthy  since  they  include  a  chronicle  of  concrete  activities  or 
happenings. 

n   The  evaluator  cannot  generalize  his/her  findings  to  other  situations  or  groups. 
Procedures 

1.  Set  the  boundaries  by  geographical  areas,  themes,  and/or  interests.  Consider  cost, 
time,  and  other  constraints. 

2.  Determine  the  unit  of  analysis  or  the  "thing"  to  be  studied. 

3-   Select  the  site(s).  Decide  whether  you  will  sample  a  group  or  evaluate  the  whole. 

4.  Establish  initial  contacts  at  the  site.  This  may  include  seeking  permission  and  gaining 
the  confidence  and  confidentiality  of  the  supervisors  and  others  involved  in  the  evalu- 
ation process. 

5 .  Develop  data  collection  procedures.  These  may  include  observation,  interview,  records 
analysis,  and/or  gathering  data  unobtrusively. 

6.  Develop  a  plan  for  organizing,  coding,  and  storing  data. 

7.  Train  staff  in  the  methodology  to  be  used. 

8.  Implement  the  data  collection  procedures  being  carefiil  to  plan  in  advance,  confirm 
inter\-iews,  and  arrange  for  obser%'ations. 

9.  Analyze  data  and  write  the  report. 
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Data  Collection  Method: 
Participant  Reaction  Questionnaire 

Definition 

The  participant  reaction  questionnaire  is  designed  to  obtain  the  opinions  of  the  panicipants 
regarding  the  various  aspects  of  a  presentation  or  training  program.  The  questionnaire  can  be 
used  during  a  training  program  (formative)  or  at  the  end  of  a  training  program  (summative).  It 
can  be  used  to  obtain  feedback  useful  for  input  and  process  evaluation. 

Characteristics 

A  well-designed  participant  reaction  questionnaire  provides  the  trainer  or  meeting  planner  with 
specific  information  for  use  in  improving  the  program.  Most  questionnaires  ask  participants  to 
provide  feedback  regarding  their  perceptions  of  the: 
□   Design  of  the  training  program, 

n   Instructional  techniques  used  by  the  instructor/trainer, 
n   Psychological  climate  set  up  by  the  instructor, 
n   Physical  comforts,  and 

n   Learning  experience  in  totality  (attitude  toward  the  experience). 
Procedures 

1.  Determine  the  information  needed.  Do  not  obtain  information  that  is  not  needed, 
e.g.,  "nice  to  know."  The  information  needed  will  be  affected  by  whether  the  form  is 
to  obtain  input  evaluation  information  or  process  evaluation  information.  Oftentimes, 
a  participant  reaction  questionnaire  obtains  data  useful  for  both  types  of  evaluation. 

2.  Decide  the  types  of  questions  to  be  asked.  Keep  in  mind  that  the  type  of  question 
will  have  a  major  affect  on  the  time  needed  to  analyze  the  responses.  Avoid  open- 
ended  questions  with  a  group  of  more  than  10  people.  Use  of  ranking  is  also  a  diffi- 
cult style  of  question  to  analyze. 

Consider  using: 
n   A  checklist, 

n   Multiple-choice  questions,  or 
n   A  rating  scale. 

3.  Develop  the  questions.  Write  questions  that  are  simple  and  straightfor^'ard.  Avoid 
questions  that  can  be  confusing  because  of  jargon,  unfamiliar  words,  or  phrasing. 
Limit  each  question  to  one  idea. 
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4.  Test  the  questions.  To  the  extent  practical,  obtain  input  from  individuals  who  could 
be  participants  in  the  training  where  the  questionnaire  will  be  used.  Revise  the  ques- 
tions until  they  are  clear. 

5.  Develop  the  completed  questionnaire.  Provide  clear  instructions  both  for  the  person 
administering  the  form  and  for  the  participants  who  will  complete  it. 

6.  Most  participant  reaction  questionnaires  are  completed  anonymously.  If  this  is  so, 
note  that  on  the  questionnaire. 

An  example  of  a  participant  reaction  questionnaire  is  provided  on  the  next  page. 
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Participant  Reaction  Questionnaire 


Workshop  Title 

Instructor 


Date 


The  Instructor 


Strongly 

Disagree 

Agree 

Strongly 

Disagree 

Agree 

1 .  Demonstrated  knowledge  of  the  content. 

□ 

□ 

□ 

□ 

2.  Showed  me  how  to  apply  the  content  to  my  job. 

□ 

□ 

□ 

□ 

3.  Was  organized. 

□ 

□ 

□ 

□ 

4.  Was  easy  to  understand. 

□ 

□ 

□ 

□ 

J  .  i^Cil  JUIloLiaLCLl  CIlLll  Liold-blli  . 

n 

□ 

n 

LJ 

LJ 

6.  Provided  a  variety  of  examples  and  demonstrations. 

n 
<_i 

Ti 

LJ 

7.  Provided  a  variety  of  opportunities  for  participants'  input. 

u 

r-\ 
U 

— 1 

LJ 

8.  Gave  participants  feedback  during  opportunities  to  practice. 

LJ 

LJ 

LJ 

0       T^f^\^^Tc^  <yf*r\  no  rf      r^o  r!/~\n  Hiinncr  fn*=>  rroirttnrr 

□ 

LJ 

LJ 

□ 

10.  Provided  summaries  of  information  periodically. 

LJ 

J 

LJ 

U 

11.  Answered  questions  from  the  participants. 

LJ 

LJ 

n 
_J 

n 
LI 

12.  Set  up  a  comfortable  climate  for  leaxning. 

n 

LJ 

n 

LJ 

i^j.  OLiiycu  Oil  ciic  Liinc  icncQUic. 

n 

LJ 

14.  Used  audio-visual  and  support  materials  appropriately. 

a 

□ 

□ 

□ 

irtp  1  j^/iT^itJCT  V  vhprfp'ffrp 

J.  fJC   J— /  tl'l-fl'sL            L/C  r  ri-^C 

Strongly 

Disagree 

Agree 

Strongly 

Disagree 

Agree 

1 !!).  1  ne  amount  or  time  was  auecjua.ce  ror  tne  material  tna.t 

LJ 

n 

LJ 

LJ 

LJ 

was  included. 

l6.  I  can  use  the  information  in  my  work. 

n 

LJ 

r\ 
>J 

LJ 

LJ 

17.  The  learning  experience  met  my  expectations. 

n 
>_i 

LJ 

LI 

LI 

XO.  X   will   LLov    LXIC   WUlI^^^ilWlJ  tJI   ^\JLLloC  lliaLCIl&ib  oo 

n 

n 

□ 

a  reference. 

19-  As  a  result  of  this  experience,  I  will  work  to  make  some 

□ 

□ 

□ 

□ 

changes  when  I  get  back  to  my  job. 

20.  I  would  recommend  this  workshop/course. 

□ 

□ 

□ 

□ 

The  Facility 

Strongly 

Disagree 

Agree 

Strongly 

Disagree 

Agree 

21.  Meeting  place  was  convenient. 

□ 

□ 

□ 

□ 

22.  Meeting  room  was  comfortable. 

□ 

□ 

□ 

□ 

23.  Meals  and/or  snacks  were  enjoyable. 

□ 

□ 

□ 

□ 

24.  I  could  see  and  hear  the  instructor. 

□ 

□ 

□ 

□ 

25.  Materials  helped  me  to  learn. 

□ 

□ 

□ 

□ 

Please  complete  the  following  statements. 

26.  The  most  positive  aspect  of  this  workshop/course  was... 


n 


Please  mark  your  response  for  items  1-25  in  the  appropriate  box.  Do  not  sign  your  name. 
Leave  your  completed  questionnaire  on  the  designated  table  at  the  front  of  the  room. 


27.  To  improve  this  workshop/course  the  next  time,  I  suggest. 


Thank  you  for  providing  this  important  feedback. 
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Definition 

Portfolio  assessment  is  one  method  of  authentic  assessment  where  learners  assemble  a  portfolio  or 
collection  of  documents  which  demonstrate  their  acquisition  and  application  of  the  knowledge 
and  skills  defined  by  the  performance  objective(s). 

Characteristics 

A  student  portfolio  may  be  an  actual  collection  of  documents  from  different  sources  or  it  may  be 
the  completion  of  several  documents  included  in  a  workbook.  It  should  include: 
□    Evidence  of  progress  toward  the  performance  objective(s), 
n    Items  that  indicate  transfer  of  learning  from  classroom  to  life  or  job, 
n   Things  that  convey  learning  activities  or  artifacts  from  lessons, 
n    Evidence  of  self-reflection  and  self-knowledge, 

n    Items  that  tell  the  story  of  the  journey,  of  the  year,  or  the  training  program, 

n   Things  that  show  growth  and  change, 

n   Items  that  demonstrate  "my  best  work  to  date"  and, 

n   Anything  the  student  wants  for  whatever  reason — after  all,  it  is  his/hers! 


Procedure 

This  type  of  authentic  assessment  can  be  used  effectively  to  document  student  performance 
(Valencia,  et  al,  1990).  However,  in  the  use  of  this  type  of  assessment  for  process  or  product  eval- 
uation, "theory  seems  far  in  advance  of  practice"  (Calfee  and  Perfumo,  1993).  It  involves  the  stu- 
dent assembling  a  collection  of  documents  that,  taken  together,  show  the  student's  work. 
Historically,  certain  professionals,  such  as  artists,  fashion  designers,  architects,  advertising  agents, 
and  others  have  assembled  a  portfolio  to  illustrate  their  body  of  work.  This  same  concept  is 
applied  in  portfolio  assessment.  Portfolio  assessment  is  authentic  in  that  it  shows  an  application 
that  focuses  on  how  an  individual  can  use  the  knowledge  and  skills  acquired. 

1 .   Have  the  student  compile  several  documents  into  a  portfolio  that  demonstrates  acqui- 
sition of  knowledge  and  skills  congruent  to  the  performance  objectives.  Such  portfo- 
lios serve  to  document  the  authentic  assessment  of  student  performance  and  help  to 
integrate  assessment  with  instruction.  The  portfolio  of  student  work  can  provide  a 
basis  for  ongoing  dialogue  between  the  teacher  and  the  student,  the  student  and  the 
parent,  the  parent  and  the  teacher,  and  subsequently  between  the  student  and  a 
teacher  in  another  school,  as  well  as,  perhaps,  between  the  student  and  an  employer. 
Many  schools  across  the  country  are,  in  fact,  moving  to  the  use  of  student  portfolios 
in  place  of  student  transcripts. 
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This  evaluation  technique  can  be  used  effectively  for  adult  training  situations  and  is 
being  used  more  frequently.  Realistically,  in  order  for  portfolio  assessment  to  be  used 
successfully  in  adult  training  experiences,  the  training  must  occur  over  a  period  of 
time  that  allows  for  the  learner  to  assemble  several  evidences  of  acquisition  and  appli- 
cation of  the  training  content.  The  design  considerations  of  portfolio  assessment  are 
the  same  for  K-12  students  and  adult  learners. 


Assess  the  assembled  portfolios  using  pre-determined  criteria  based  on  performance 
objectives  (measurable  outcomes).  The  individual  responsible  for  the  assessment  of  the 
student-assembled  portfolio  must  be  given  a  proper  orientation  and  training  experi- 
ence. The  assessment  of  the  portfolios  in  K-12  classrooms  may  involve  the  teacher, 
the  student,  classmates,  parents,  community  members,  or  known  experts.  The  assess- 
ment of  the  portfolios  in  adult  training  may  include  the  instructor,  the  adult  learner, 
peers,  a  supervisor,  or  known  content  experts. 

One  of  the  concerns  about  using  this  type  of  assessment  as  a  method  of  process  or 
product  evaluation  is  the  issue  of  validity  and  reliability.  Validity  can  be  assumed 
because  this  assessment  process  is  based  on  student  assembled  portfolios  demonstrat- 
ing authentic  application  of  new  knowledge  and  skills.  However,  reliability  is  a  dif- 
ferent matter.  In  one  study  reported  with  language  teachers,  a  panel  was  used  to 
establish  reliability  (Calfee  and  Perfumo,  1993).  Two  or  more  teacher- judges  scored 
each  student  portfolio  to  establish  consistency  with  the  stated  performance  objectives. 
Calfee  and  Perfumo  report  that  this  practice  is  rare.  When  portfolio  assessment  is 
used  in  the  classroom  or  a  training  situation,  the  instructional  designer  should  pay 
careful  attention  to  the  issue  of  reliability. 

Schools  report  using  portfolio  assessment  as  one  of  several  measures  of  student 
achievement.  Such  an  approach  is  also  appropriate  for  use  in  adult  training  situations. 
It  was  used  in  the  design  of  a  five-day  orientation  program  for  new  school  food  service 
managers  in  Mississippi  titled,  DECIDE  to  Succeed:  An  Orientation  for  New  Managers 
(1995,  Office  of  Child  Nutrition,  Mississippi  Department  of  Education)  and  is  also 
being  used  to  assess  teachers  in  a  national  program. 
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Sampling  Protocol  for 
NET  Needs  Assessment  Surveys 

Prepared  for  the  14-State  NET  Needs  Assessment  Consortium 

by 

Jack  Hamilton.  Ph.D. 
Lumina  Training  Associates,  Birmingham,  Alabama 
November  1994 

Introduction 

A  sample  is  a  subset  of  a  population  observed  for  purposes  of  making  inferences  about  that  popula- 
tion. The  quality  of  a  sample  is  based  on  the  degree  to  which  it  is  representative  of  the  population.  A 
sample  is  considered  to  be  representative  if  the  characteristics  of  the  sample  are  generally  the 
same  as  those  of  the  population  from  which  it  was  selected. 

Although  there  are  a  varien,"  of  sampling  procedures  available  to  researchers,  the  discussion  which 
follows  is  limited  to  probability  sampling.  The  basic  principle  of  probability  sampling  is  that  a 
sample  will  be  representative  of  the  population  from  which  it  is  selected  if  all  members  of  the 
population  have  an  equal  chance  of  being  selected  in  the  sample.  It  is  helpful  to  think  of  proba- 
bility sampling  as  involving  four  separate  steps: 

1)  identification  of  the  population  and  selection  of  an  appropriate  sampling  frame, 

2)  determination  of  the  sampling  procedures  to  be  used, 

3)  determination  of  the  minimum  sample  size  acceptable  for  the  research  purpose,  and 

4)  selection  of  the  sample. 

It  should  be  noted  however,  that  there  is  considerable  variation  available  within  each  step 
depending  on  the  number  and  distribution  of  the  populations  to  be  sun^eyed. 

Populations  and  Sampling  Frames 

The  researcher  must  begin  with  a  careful  specification  of  the  population  to  be  sampled,  such  as 
the  population  of  school  food  service  district  directors.  Once  this  population  has  been  identified, 
the  researcher  must  obtain  or  create  a  sampling  frame.  The  sampling  frame  is  an  actual  list  of  the 
elements  or  individuals  in  the  population  from  which  the  sample  will  be  selected. 

The  characteristics  of  the  sampling  frame  are  largely  determined  by  the  form  and  extensiveness  of 
available  records.  The  simplest  samphng  frame  is  an  available  list  of  all  members  of  the  popula- 
tion. The  State  Department  of  Education  Directory  usually  provides  a  list  of  the  school  food  ser- 
vice district  directors  and  can  be  used  as  a  sampling  frame  for  this  population.  This  same  directo- 
ry also  provides  a  list  of  school  districts  and  schools,  and  thus  can  be  used  as  the  sampling  frame 
for  other  NET-targeted  populations.  The  State  Agency*  which  administers  the  Child  Nutrition 
Programs  has  lists  of  other  populations  including  the  Child  and  Adult  Care  Food  Program  popu- 
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lacions.  An  available  list  of  all  members  of  a  population  can  be  used  to  select  a  sample  in  a  single 
stage.  However,  when  such  a  list  is  unavailable  or  non-existent  the  researcher  must  create  such  a 
list  from  other  available  information,  often  involving  several  different  stages  in  the  selection  of 
the  sample.  It  should  be  noted  that  probability  sampling  is  not  possible  if  a  sampling  frame  does 
not  exist  or  cannot  be  created. 

Sampling  Procedures 

Two  requirements  must  be  met  in  order  to  select  a  probability  sample:  (1)  the  availabiliry  of  a 
complete  list  of  the  elements  or  persons  in  the  population,  and  (2)  the  consistent  application  of  ran- 
domization procedures. 

A  simple  random  sample  can  be  utilized  when  the  sampling  frame  is  comprised  of  a  complete  list  of 
the  population  to  be  sampled.  When  such  a  list  is  available,  the  researcher  can  draw  a  simple  ran- 
dom sample  in  one  of  two  ways.  First,  if  the  sampling  frame  is  on  a  computer,  then  a  computer 
program  can  be  written  to  generate  a  random  sample  directly  from  the  list.  Secondly,  if  the  sam- 
pling frame  is  only  available  on  a  hard  copy  list,  the  researcher  can  assign  numbers  to  each  ele- 
ment of  the  population,  and  select  the  sample  by  referring  to  a  "Table  of  Random  Numbers."  (A 
table  of  random  numbers  can  be  generated  by  computer,  or  the  researcher  can  find  such  tables  in 
the  appendices  to  most  textbooks  in  statistics  and  social  research.) 

The  selection  of  a  simple  random  sample  requires  the  researcher  to:  (1)  assign  a  numerical  identi- 
fication to  each  element  of  the  population,  (2)  specify  the  sample  size,  and  (3)  use  random  num- 
bers which  correspond  to  the  assigned  numbers  to  select  the  sample.  Given  a  list  of  100  CACFP 
sponsors,  for  example,  a  sample  of  20  sponsors  can  be  easily  selected. 

First,  the  researcher  assigns  each  sponsor  a  sequential  number  from  one  to  100.  Secondly,  the 
researcher  obtains  twenty  random  numbers  between  one  and  100  by:  (1)  generating  twenty  dif- 
ferent random  numbers,  each  between  one  and  100,  by  computer,  or  (2)  selecting  the  first  twenty 
numbers  between  one  and  100  from  a  table  of  random  numbers.  Thirdly,  the  researcher  selects 
the  sample  of  twenty  by  matching  each  random  number  with  the  number  assigned  to  the  popula- 
tion. Thus,  if  the  first  random  number  is  55,  the  55th  person  on  the  list  of  sponsors  is  selected. 

Although  the  use  of  a  table  of  random  numbers  is  quite  simple,  the  actual  process  of  selecting  a 
simple  random  sample  can  prove  to  be  tedious  and  laborious,  especially  if  the  population  listed 
on  the  sampling  frame  is  quite  large.  Consequently,  systematic  random  sampling  is  used  and  pre- 
ferred by  most  researchers. 

Systematic  random  sampling  recognizes  that  a  systematic  sample  will  be  equivalent  to  a  simple  ran- 
dom sample,  if  the  sampling  frame  used  is  essentially  random.  To  select  a  systematic  random 
sample  the  researcher  must: 

1)  specify  the  sample  size, 

2)  determine  a  sampling  ratio  (sample  size  divided  by  the  population  size),  and 

3)  randomly  select  the  first  element  or  person. 
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For  example,  a  researcher  who  wishes  to  survey  10  people  from  a  population  of  100  people  will 
have  a  1/1 0th  sampling  ratio.  Consequently,  all  the  researcher  must  do  is  randomly  select  the  first 
case  (perhaps  using  a  table  of  random  numbers),  and  thereafter  select  every  tenth  name  of  the  list. 

In  many  studies  the  researcher  wants  to  assure  that  different  kinds  of  conditions  or  groups  are 
also  represented  in  the  sample.  In  such  situations  the  sampling  procedure  must  be  modified  to 
assure  that  the  sample  includes  representatives  of  such  groups  ("strata").  Thus,  whether  the 
researcher  is  drawing  a  simple  random  sample  or  a  systematic  random  sample,  the  sample  proce- 
dure may  include  the  selection  of  a  stratified  random  sample. 

In  a  stratified  random  sample  the  researcher  divides  the  individuals  or  groups  into  categories,  and 
then  independent  samples  are  selected  from  each  group  or  strata.  Proportional  stratified  sam- 
pling, in  which  the  sample  from  each  group  is  proportional  to  the  size  of  the  group  in  the  popu- 
lation, is  generally  preferred  in  order  to  assure  a  more  representative  sample  than  would  be 
obtained  by  use  of  only  a  random  sample  or  a  systematic  random  sample.  Proportional  stratified 
sampling  is  appropriate  for  most  NET  needs  assessment  surveys.  The  student  enrollment  figures 
for  school  districts  can  be  used  to  determine  the  strata  for  samples  of  school  food  service  person- 
nel, principals,  curriculum  directors,  or  other  groups  in  schools. 

Determining  Sample  Size 

The  researcher  should  he  aware  that  the  size  of  the  population  has  relatively  no  relationship 
to  the  size  of  the  probability  sample  to  be  selected.  Instead,  the  size  of  the  sample  is  largely  a 
function  of  the  desired  level  of  accuracy.  Although  a  discussion  of  probability  theory  is  beyond 
the  scope  of  this  overview,  an  understanding  of  a  few  basic  assumptions  are  of  value  to  an  under- 
standing of  statistical  analysis  and  the  determination  of  sample  size. 

Many  of  the  statistical  procedures  for  determination  of  sample  size  as  well  as  data  analysis  are 
derived  from  the  concept  of  a  normal  distribution.  The  normal  distribution  is  commonly  depicted 
as  the  traditional  and  perfect  "bell-shaped  curve."  However,  this  distribution  is  a  theoretical  distri- 
bution which  is  defined  only  by  mathematical  rules.  In  this  theoretical  normal  distribution  the 
horizontal  axis  is  continuous  and  represents  all  possible  values  between  plus  (+)  and  minus  (-) 
infinity.  The  vertical  axis  represents  the  probability  density  for  each  possible  value. 

The  conception  of  a  normal  distribution  permitted  statisticians  to  mathematically  construct  stan- 
dardized tables  of  probabilities.  Further,  the  existence  of  a  standardized  normal  distribution  permit- 
ted statisticians  to  derive  mathematical  equations  which  provided  for  a  variety  of  ways  to  convert 
sample  data  to  standardized  measures  which,  when  compared  with  the  standardized  normal  distribu- 
tion permits  the  researcher  to  identify  the  probability  of  obtaining  such  results  from  chance 
expectation. 

The  normal  distribution  is  an  excellent  approximation  of  the  sampling  distribution  of  the  popu- 
lation mean.  In  addition,  the  sampling  distribution  of  the  sample  mean  approaches  a  normal  dis- 
tribution as  the  sample  size  increases.  Thus,  with  a  sufficiently  large  sample,  the  normal  distribu- 
tion provides  an  excellent  approximation  to  the  exact  sampling  distribution. 
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A  major  advantage  of  probability  sampling  is  that  probability  theory  permits  the  researcher  to 
estimate  the  representativeness  of  the  sample.  Any  sample  estimate  may  include  some  margin  of 
error.  This  possible  error  is  typically  referred  to  as  the  sampling  error,  and  refers  to  the  statistical 
error  which  may  arise  from  any  effort  to  represent  a  population  with  a  sample.  A  description  of 
the  sampling  error  for  a  given  sample  contains  two  measures,  the  confidence  level  and  the  confi- 
dence interval. 

The  confidence  interval  refers  to  the  degree  of  sampling  error  that  the  researcher  is  willing  to 
accept.  A  sampling  error  of  5%,  for  example,  means  that  the  error  may  be  as  much  as  5%  greater 
than,  or  5%  less  than,  the  true  population  mean.  This  sampling  error  can  be  expressed  as  a  confi- 
dence interval  ranging  from  +5%  to  -5%. 

The  confidence  level  is  used  in  conjunction  with  a  specified  confidence  interval,  and  serves  as  a  mea- 
sure of  the  degree  of  confidence  in  the  likelihood  that  the  sample  is  within  the  specified  error 
range.  Statistically,  the  95%  confidence  level  (.95)  means  that  if  a  researcher  selected  100  differ- 
ent samples,  95  of  the  samples  would  fall  within  the  specified  confidence  level. 

Table  1.,  "Minimum  Sample  Size  and  Sampling  Error,  by  Confidence  Level  and  Confidence 
Interval"  which  follows,  is  provided  to  illustrate  how  sampling  error  is  related  to  sample  size.  As 
Table  1.  illustrates,  a  researcher  who  is  willing  to  accept  the  conventional  95%  confidence  level 
that  the  sampling  error  is  within  a  4%  sampling  error  would  need  to  select  a  sample  of  600  peo- 
ple. In  order  to  be  99%  confident  that  the  sample  is  within  4%  of  the  population  mean,  the 
researcher  would  need  to  increase  the  sample  size  to  1,037.  The  specified  sample  size,  in  either 
case,  would  be  appropriate  whether  the  population  being  sampled  was  a  city,  a  state,  or  the 
United  States  of  America.  (The  Gallup  Poll,  for  example,  typically  interviews  about  1,200  people 
as  an  estimate  of  a  population  in  excess  of  200-million  people.  Table  1.  reveals  that  Gallup  can  be 
95%  confident  that  the  sample  mean  is  within  a  3%  sampling  error  of  the  true  population 
mean.) 
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Table  1. 

Minimum  Sample  Size  and  Sampling  Error 
by  Confidence  Level  and  Confidence  Interval 

Acceptable  Acceptable  Confidence  Level 

Confideyice 


Intenal 

.95 

.99 

+  1%  to -\7c 

9,604 

16,587 

+2%  to  -2% 

2,401 

4,147 

+3%  to  -0% 

1,067 

1,843 

+4%  to  -A% 

600 

1,03" 

+5%  to  -5% 

384 

663 

+6%  to  -6% 

26- 

461 

+7%  to  -7% 

196 

339 

SOURCE:  Adapted  from  Charles  Backstrom  and  Gerald  Hursh.  (1963).  Survey  Research.  i^finneapoLis:  Northwestern 
University  Press,  pp.  28-53- 

It  IS  important  to  remember  that  the  size  of  the  sample  is  largely  a  function  of  the  desired  level  of 
accuracy,  and  is  not  greatly  influenced  by  the  size  of  the  population.  The  sample  sizes  reported  in 
Table  1.  reveal  that,  regardless  of  the  size  of  the  population,  extremely  large  samples  are  rarely 
needed.  In  fact,  although  generally  considered  too  small  for  such  precise  and  rigorous  statistical 
analysis  as  required  in  the  formal  testing  of  a  specified  hypothesis,  the  researcher  who  draws  a 
probabilin,-  sample  of  100  people  can  be  95%  confident  that  the  results  are  within  a  10%  sam- 
pling error.  Since  NET  needs  assessment  data  are  most  likely  to  be  used  to  make  general  manage- 
ment decisions,  a  NET  researcher  may  tolerate  a  larger  sampling  error  and  be  comfortable  with  a 
smaller  sample  size.  It  is  recommended  for  SET  needs  assessment  surteys  that,  whenever  the 
population  group  includes  200  people  or  less,  the  entire  population  be  included  in  the  survey. 
The  use  of  the  entire  population  eliminates  all  sampling  error  and,  of  course,  makes  it  unnec- 
essary to  select  a  sample  from  that  group. 

Selecting  the  Sample 

The  overview  of  the  first  three  steps  in  the  sampling  process  reveals  that  the  process  of  selecting  a 
sample  can  be  undertaken  in  a  variety-  of  ways.  However,  once  the  researcher  identifies  the  sam- 
pling frame,  defines  the  sampling  procedure,  and  specifies  the  size  of  the  sample,  then  the  acnoal 
selection  of  the  sample  is  quite  simple.  Beginning  with  a  list  of  the  population  (the  sampling 
frame),  the  researcher  merely  foUows  those  procedures  in  order  to  prepare  a  list  of  the  sample  popu- 
lation. 
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The  sampling  procedures  described  above  suggest  that  the  preferred  sampling  procedure  for 
the  NET  needs  assessment  surveys  consist  of  the  selection  of  a  systematic  stratified  random 
sample.  However,  in  order  to  simplify  the  description  of  the  sampling  procedures,  the  overview 
of  the  four  steps  was  presented  in  terms  of  a  single  sample.  Unfortunately,  the  selection  of  the 
sample  does  not  always  proceed  as  smoothly  as  the  four  steps  indicate.  Consequently,  there  are 
several  useful  considerations  that  may  help  the  researcher  deal  with  certain  problems  as  they  arise. 

First,  it  is  helpful  to  begin  a  survey  design  with  an  awareness  that  the  researcher  will  often  be 
drawing  a  series  of  samples  rather  than  a  single  sample.  This  may  occur  as  a  result  of  a  number  of 
factors,  including  the  possibility  that: 

1)  different  groups  or  strata  may  be  found  within  the  population, 

2)  each  such  group  may  have  a  different  sampling  frame  in  a  different  format, 

3)  the  questionnaires  to  be  administered  may  not  be  identical  for  each  organizational  unit,  and 

4)  in  many  cases  the  data  collected  for  each  group  within  the  sample  will  be  coded  and  record- 
ed in  a  separate  file  for  purposes  of  data  analysis. 

Similar  agencies  in  different  States,  as  well  as  different  agencies  in  the  same  State,  do  not  always 
collect  the  same  information  on  the  same  form.  Thus,  for  example,  in  attempting  to  select  a  sam- 
ple of  "School  Food  Service  (SFS)  Managers,"  the  researcher  in  "State  A"  may  be  fortunate  enough 
to  find  that  the  State  Agency  has  a  list  of  the  names  and  addresses  of  the  SFS  Managers  in  every 
school  district.  A  random  sample  of  SFS  Managers  can  be  drawn  directly  from  such  a  list. 

In  contrast,  a  researcher  in  "State  B"  may  find  that  there  is  no  list  of  all  the  SFS  Managers  in  the 
State.  Instead,  the  available  data  are  in  the  form  of  a  list  of  all  public  and  private  schools  and 
organizations  that  prepare  food  under  the  Child  Nutrition  Program.  This  is  a  more  complicated 
situation,  and  will  require  the  researcher  to  indirectly  select  a  sample  of  SFS  Managers  by  select- 
ing a  sample  of  school  districts  and  schools. 

Secondly,  it  should  be  noted  that  differences  in  the  available  sampling  frames  not  only  influence 
the  way  the  sample  is  selected,  but  will  also  affect  procedures  for  administering  the  survey.  In  the 
two  situations  described  above,  the  researcher  in  "State  A"  can  mail  the  survey  questionnaires 
directly  to  the  SFS  Managers,  but  the  researcher  in  "State  B"  must  mail  the  questionnaire  to  the 
school  district  Food  Service  Directors,  relying  upon  them  to  distribute  the  questionnaire  to  the 
Managers. 

Thirdly,  the  researcher  must  realize  that  the  characteristics  of  the  available  sampling  frame  or  list 
will  also  influence  the  selection  of  the  sampling  procedures  to  be  used.  More  importantly,  the 
researcher  should  be  aware  that  a  different  sampling  procedure  may  be  used  with  different  popu- 
lations. Thus,  were  one  stratum  or  group  has  an  extremely  large  population,  a  systematic  random 
sample  might  be  selected.  This  procedure,  for  example  would  be  suited  to  the  selection  of  a  sam- 
ple of  Family  Day  Care  Home  Providers.  In  contrast,  where  another  group  has  an  extremely  small 
population,  a  simple  random  sample  might  be  preferred.  This  procedure  would  be  more  appro- 
priate for  the  selection  of  a  sample  of  Child  Care  Center  Sponsors. 

Fourthly,  although  a  stratified  sample  is  preferred,  the  researcher  must  be  aware  that  the  same 
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strata  may  be  defined  differently  for  various  groups  within  the  population.  The  researcher  may  be 
interested  in  differences  ber^-een  public  schools  and  private  organizations  that  provide  meals.  The 
researcher  may  also  be  interested  in  differences  with  respect  to  the  number  of  meals  served,  dis- 
tinguishing between  large,  medium,  and  small  programs  based  on  the  number  of  meals  served.  In 
public  school  Child  Nutrition  Programs,  "large"  may  be  defined  as  over  1,000  meals,  whereas  in 
child  care  centers,  "large"  may  be  defined  as  over  50  meals. 

The  research  problem  in  this  situation  is  the  recognition  that  no  one  definition  of  the  strata  is 
appropriate  for  all  populations  being  surveyed.  The  best  advice  to  the  researcher  in  such  a  situa- 
tion is  to  apply  to  each  population:  (1)  the  same  definition  or  categories  for  each  strata  examined 
(such  as  small,  medium,  and  large),  and  (2)  equal  intervals  of  measurement  for  each  categor}-, 
based  on  the  largest  and  smallest  numbers  for  the  respective  population. 

Nevertheless,  despite  such  difficulties,  the  use  of  stratified  sample  ser\-es  to  improve  the  represen- 
tativeness of  the  sample  because  it  reduces  the  degree  of  sampling  error.  There  are  two  potential 
sources  of  variation  in  a  random  sample.  Sampling  error  may  be  found  uithin  each  strata  as  well 
as  between  strata.  The  stratified  sample  ser\'es  to  effectively  eliminate  the  sampling  error  between 
strata. 

Finally,  when  confronted  by  a  problem  situation  that  requires  a  decision  about  the  sampling 
process,  the  researcher  should  turn  to  the  experts  in  survey  research  and  statistical  analysis. 
Numerous  texts  are  readily  available  to  assist  the  researcher,  and  a  few  examples  are  provided  in 
the  Selected  Bibliography  for  this  paper.  Any  problems  in  sample  selection  that  cannot  be  readily 
resolved  from  the  available  and  extensive  literature,  should  be  resolved  by  the  principle  of  ran- 
dom selection.  Such  questions  are  best  solved  by  "flipping  a  coin"  because  only  a  random  decision 
is  free  of  the  possible  bias  of  the  researcher 

Selected  Bibliography 

Numerous  books  and  articles  are  available  to  assist  in  all  stages  of  research.  The  beginning 
researcher  may  consult  an  introductory  textbook  in  research,  such  as  Earl  Babbie,  Suney  Research 
Methods  (Wadsworth,  2nd  ed.,  1990),  or  The  Practice  of  Social  Research  (Wadsworth,  6th  ed.,  1992), 
both  of  which  include  a  Table  of  Random  Numbers  in  the  Appendix.  Researchers  interested 
specifically  in  the  procedures  for  selecting  a  sample  might  refer  to  the  classic  works  of  Leslie 
Kish,  Survey  Sampling  (New  York:  John  Wiley,  1965);  Hubert  Blalock,  Social  Statistics 
(Minneapolis:  Northwestern  University  Press,  1963).  The  advanced  researcher  who  wishes  to 
understand  the  intricacies  of  statistical  analysis  and  probabilit}-  theor\-,  can  trace  the  mathematical 
derivations  of  most  statistical  equations  in  William  Hays,  Statistics  (New  York:  Holt,  Rinehart, 
and  Winston,  1963). 

Copyright  ©  1994  by  C-NET  Needs  Assessment  Consortium 
(Used  with  permission.) 
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Excerpts  from 
The  Strategic  Plan  for  Nutrition  Education 

United  States  Department  of  Agriculture 
Food  and  Nutrition  Service 

Nutrition  and  Technical  Services  Division 
1993 

Edited  by:  R.  Jane  Mandell 

The  Strategic  Direction 

In  the  year  2000,  nutrition  education  will  be  a  major  educational  comp>onent  of  all  child  nutri- 
tion programs  and  offered  in  all  schools,  child  care  facilities,  and  summer  sites  as  part  of  health 
education,  as  part  of  an  interdisciplinarv"  curriculum,  or  through  other  means.  Nutrition  educa- 
tion will  be  supported  and  promoted  by  State  and  local  personnel  working  in  partnership  with 
the  children  and  their  families  as  part  of  the  total  school  environment  or  child  care  setting. 
Depending  on  available  resources,  State  agencies  will  conduct  formal  needs  assessments  and  eval- 
uations as  the  basis  for  program  development,  improvement,  and  accountabilir\'.  Nutrition  educa- 
tion staff  at  State  offices,  USDA  Headquarters,  and  USDA  Regional  offices  will  have  specific 
roles  and  responsibilities  in  the  administration  of  the  nutrition  education  program.  N~ET  will 
have  extensive  alliances  and  partnerships  with  public  and  private  concerns  at  the  local.  State,  and 
national  levels.  Through  these  relationships,  NET  will  obtain  funding  and  resources,  identiiH' 
research  agencies  in  nutrition  education  and  food  management,  identify,'  technical  assistance 
resources,  develop  evaluation  strategies,  and  work  cooperatively  with  all  groups  that  promote 
nutrition  education  and  service  of  nutritious  meals  and  snacks  for  our  nation's  children. 

The  Goals 

Nutrition  Education  and  Training 
Goal  1 

Increase  to  at  least  ~5%  the  proportion  of  the  Nation's  schools  that  provide  nutrition 
education  as  part  of  health  education,  as  part  of  an  interdisciplinary  nutrition  education 
curriculum,  or  through  other  means  to  children  and  parents  by  the  year  2000  as  docu- 
mented by  data  submitted  to  EN'S  or  by  other  sampling  methods. 

Goal  2 

Increase  to  a  least  50%  the  proportion  of  child  care  and  summer  program  facilities/sites 
that  provide  nutrition  information/education  to  children,  caregivers  or  parents  by  the 
year  2000  as  documented  by  data  submitted  to  FNS  or  by  other  sampling  methods. 
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Goal  3 

Increase  to  a  least  50%  the  proportion  of  food  service  personnel/providers  in  Child 
Nutrition  Programs  who  have  received  nutrition  information  and/or  training  (includ- 
ing food  service  training)  by  the  year  2000  as  documented  by  data  submitted  to  FNS 
or  by  other  sampling  methods. 

Nutritious  Meal  Service 

Goal  4 

Increase  to  at  least  90%  the  proportion  of  school  lunch  and  breakfast  services  with 
menus  that  are  consistent  with  the  nutrition  principles  in  the  Dietar)'  Guidelines  for 
Americans  by  the  year  2000  as  documented  by  data  submitted  to  FNS  or  by  other  sam- 
pling methods. 

Goal  5 

Increase  to  90%  the  proportion  of  child  care  and  summer  food  services  with  menus 
that  are  consistent  with  the  nutrition  principles  in  the  Dietary  Guidelines  for 
Americans  by  the  year  2000  as  documented  by  data  submitted  to  FNS  or  by  other 
sampling  methods. 

Leadership 
Goal  6 

Increase  to  at  least  75%  the  proportion  of  States  that  incorporate  applicable  goals  and 
strategies  from  the  NET  Strategic  Plan  into  their  NET  State  Plans  by  the  year  2000  as 
documented  by  data  submitted  to  FNS. 

Goal  7 

Increase  to  at  least  75%  the  proportion  of  States  that  use  national  guidelines  to  conduct 
needs  assessments  by  the  year  2000  as  docvimented  by  data  submitted  to  FNS. 

Goal  8 

Increase  to  at  least  75%  the  proportion  of  States  that  use  national  guidelines  to  evalu- 
ate NET  projects  by  the  year  2000  as  documented  by  data  submitted  to  FNS. 

Goal  9 

Increase  to  at  least  75%  the  proportion  of  nutrition  education  personnel  at  the  State, 
regional  and  national  levels  with  roles  and  responsibilities  that  are  consistent  with 
national  guidelines  by  the  year  2000  as  documented  by  data  submitted  to  FNS. 

Goal  10 

Establish  partnerships  with  at  least  10  allied  nutrition,  health,  food  management,  or 
education  organizations;  public  or  private  agencies  or  groups;  or  others  interested  in 
child  nutrition  issues  by  the  year  2000. 
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Attitude  A  feeling  or  emotion  toward  a  fact  or  state. 

Audience  The  group  of  participants  who  are  targeted  for  a  program  or  project. 
Context  The  situation,  the  environment,  the  miheu. 

Context  evaluation  Evaluation  of  the  situation  or  environment  in  which  a  program  or  project 
operates. 

Criteria  Standards  for  making  a  qualitative  or  quantitative  judgment. 

Data  collection  method  A  technique  used  to  obtain  data.  The  data  may  be  used  for  needs 
assessment  or  for  evaluation. 

Descriptive  statistics  Statistics  that  help  describe  certain  characteristics  of  a  research  sample. 
Two  basic  types  of  measures  used  are  measures  of  central  tendency  and  variability'  (dispersion). 

Evaluation  plan  The  systematic,  written  procedure  describing  how  each  State  Plan  objective,  or 
program/project  objective  will  be  evaluated.  The  plan  should  include  evaluation  questions,  the 
data  to  be  collected,  the  standard  to  assess  acceptability,  the  evaluation  method/procedure  to  be 
used,  a  work  schedule,  and  an  identification  of  responsibility. 

Evaluation  type  One  of  the  five  types  of  evaluation  comprising  the  CIPPI  model:  context, 
input,  process,  product,  and  impact. 

Formative  evaluation  Evaluation  which  occurs  during  the  planning  and  implementation  of  a 
program  or  project.  Context,  input,  and  process  evaluation  may  be  used  formatively.  In  fact, 
using  these  three  types  of  evaluation  as  formative  evaluation  provides  the  program  planner  data 
on  which  to  base  decisions  to  make  corrections  in  a  program  or  project  during  the  planning  and 
implementation  phase,  rather  than  waiting  until  it  is  completed. 

Frequency  distribution  Grouping  a  large  mass  of  data  into  different  classes  and  then  determin- 
ing the  number  of  observations  that  fall  in  each  of  the  classes.  An  arrangement  of  data  in  tabular 
form  is  called  a  frequency  distribution. 

Impact  evaluation  Evaluation  of  the  cumulative  differences  a  program  or  project  makes;  "the 
bottom  line." 
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Inferential  statistics  Comprises  those  methods  concerned  with  the  analysis  of  a  subset  of  data 
leading  to  predictions  or  inferences  about  the  entire  set  of  data. 

Input  evaluation  Evaluation  which  assesses  the  appropriateness  and  adequacy*  of  all  resources 

that  go  mto  the  design  and  implementation  of  a  program  or  project. 

Mean  A  measure  of  central  tendency-  which  is  determined  by  the  simple  arithmetic  average. 

Median  A  measure  of  central  tendency  that  is  equivalent  to  the  50th  percentile. 

Mode  A  measure  of  central  tendency'  that  is  the  most  frequently  observed  measure. 

Opinion  A  generally  held  view;  a  belief  stronger  than  an  impression  and  less  strong  than  posi- 
tive knowledge. 

Perception  A  mental  image;  discernment. 

Population  The  total  group  or  total  number  of  observations  with  which  a  researcher  is 
concerned. 

Process  A  course  of  action  or  a  series  of  steps  leading  toward  an  end. 

Process  evaluation  Evaluation  which  provides  information  about  program  implementation  and 
whether  learners  are  meeting  interim  performance  objectives. 

Product  evaluation  Evaluation  which  focuses  on  the  attainment  of  performance  objectives  after 

the  completion  of  the  program. 

Prospective  Relating  to  or  effective  in  the  future. 
Proximity  Soon  forthcoming,  very  near. 
Retrospective  Relating  to  or  affecting  things  past. 
Qualitative  data  Data  which  are  categorical. 
Quantitative  data  Data  which  are  numerical. 
Sample  A  subset  of  a  population. 

Summative  evaluation  Evaluation  used  at  the  end  of  a  program  or  project.  Context,  input, 
process,  product,  and  impact  evaluation  may  be  used  as  summative  evaluation  after  a  program  or 
project  has  been  completed. 
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